Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:08.559819 22661 test_util.cc:268] Using random seed: -470839907
I20240501 13:59:08.571552 22661 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:08.571641 22661 ts_itest-base.cc:113] --------------
I20240501 13:59:08.571682 22661 ts_itest-base.cc:114] 5 tablet servers
I20240501 13:59:08.571724 22661 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:08.571763 22661 ts_itest-base.cc:116] --------------
2024-05-01T13:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:08Z Disabled control of system clock
I20240501 13:59:08.597574 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:38743
--webserver_interface=127.22.33.126
--webserver_port=0
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.33.126:38743 with env {}
W20240501 13:59:08.936692 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:08.937090 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:08.937197 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:08.944924 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:08.945080 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:08.945130 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:08.945187 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:08.954600 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.33.126:38743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.33.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:08.955976 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:08.957751 22673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:08.969957 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:08.970470 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:08.970604 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:08.971410 22673 server_base.cc:1034] running on GCE node
I20240501 13:59:08.972318 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:08.973945 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:08.975219 22673 hybrid_clock.cc:648] HybridClock initialized: now 1714571948975139 us; error 58 us; skew 500 ppm
I20240501 13:59:08.975634 22673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:08.978330 22673 webserver.cc:460] Webserver started at http://127.22.33.126:41905/ using document root <none> and password file <none>
I20240501 13:59:08.979128 22673 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:08.979271 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:08.979611 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:08.983160 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "575636b8cd694bc08f4ccc4666dcf4bc"
format_stamp: "Formatted at 2024-05-01 13:59:08 on dist-test-slave-3610"
I20240501 13:59:08.984012 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "575636b8cd694bc08f4ccc4666dcf4bc"
format_stamp: "Formatted at 2024-05-01 13:59:08 on dist-test-slave-3610"
I20240501 13:59:08.989905 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:08.993813 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:08.995442 22673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:08.995647 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
uuid: "575636b8cd694bc08f4ccc4666dcf4bc"
format_stamp: "Formatted at 2024-05-01 13:59:08 on dist-test-slave-3610"
I20240501 13:59:08.995860 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:09.011128 22673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:09.012048 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:09.012352 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:09.030045 22673 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.126:38743
I20240501 13:59:09.030068 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.126:38743 every 8 connection(s)
I20240501 13:59:09.032166 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:09.036213 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 22673
I20240501 13:59:09.036482 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:09.042297 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:09.056414 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc: Bootstrap starting.
I20240501 13:59:09.060791 22739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:09.061945 22739 log.cc:824] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:09.064855 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc: No bootstrap required, opened a new log
I20240501 13:59:09.072405 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } }
I20240501 13:59:09.072772 22739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:09.072880 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575636b8cd694bc08f4ccc4666dcf4bc, State: Initialized, Role: FOLLOWER
I20240501 13:59:09.073480 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } }
I20240501 13:59:09.073684 22739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:09.073805 22739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:09.073958 22739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:09.076001 22739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } }
I20240501 13:59:09.076490 22739 leader_election.cc:302] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 575636b8cd694bc08f4ccc4666dcf4bc; no voters:
I20240501 13:59:09.077169 22739 leader_election.cc:288] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:09.077317 22744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:09.077980 22744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [term 1 LEADER]: Becoming Leader. State: Replica: 575636b8cd694bc08f4ccc4666dcf4bc, State: Running, Role: LEADER
I20240501 13:59:09.078575 22744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } }
I20240501 13:59:09.079310 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:09.081804 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 575636b8cd694bc08f4ccc4666dcf4bc. Latest consensus state: current_term: 1 leader_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } } }
I20240501 13:59:09.082115 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:09.082636 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575636b8cd694bc08f4ccc4666dcf4bc" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38743 } } }
I20240501 13:59:09.082978 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:09.083257 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:09.094333 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:09.106058 22752 catalog_manager.cc:1347] Generated new cluster ID: 400395b50de74854a4f2eb81df6f8cad
I20240501 13:59:09.106191 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:09.121719 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:09.123154 22752 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:09.131705 22752 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 575636b8cd694bc08f4ccc4666dcf4bc: Generated new TSK 0
I20240501 13:59:09.132629 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:09.151396 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.65:0
--local_ip_for_outbound_sockets=127.22.33.65
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 13:59:09.508534 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:09.508843 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:09.508940 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:09.516438 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:09.516629 22764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.65
I20240501 13:59:09.525732 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.65
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:09.527048 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:09.528761 22764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:09.540683 22769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.540277 22770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.541108 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:09.542304 22764 server_base.cc:1034] running on GCE node
I20240501 13:59:09.542886 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:09.543864 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:09.545095 22764 hybrid_clock.cc:648] HybridClock initialized: now 1714571949545028 us; error 41 us; skew 500 ppm
I20240501 13:59:09.545390 22764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:09.547600 22764 webserver.cc:460] Webserver started at http://127.22.33.65:44099/ using document root <none> and password file <none>
I20240501 13:59:09.548323 22764 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:09.548445 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:09.548739 22764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:09.551329 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "170e490f2fc44012a11802033eccc5f9"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:09.552088 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "170e490f2fc44012a11802033eccc5f9"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:09.557248 22764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:09.560673 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:09.562147 22764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:09.562285 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "170e490f2fc44012a11802033eccc5f9"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:09.562446 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:09.582262 22764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:09.583089 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:09.583379 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:09.584290 22764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:09.585834 22764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:09.585907 22764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:09.586000 22764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:09.586043 22764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:09.617723 22764 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.65:33589
I20240501 13:59:09.617764 22890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.65:33589 every 8 connection(s)
I20240501 13:59:09.619033 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:09.622262 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 22764
I20240501 13:59:09.622476 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:09.626269 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.66:0
--local_ip_for_outbound_sockets=127.22.33.66
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:09.634092 22891 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38743
I20240501 13:59:09.634464 22891 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:09.635326 22891 heartbeater.cc:511] Master 127.22.33.126:38743 requested a full tablet report, sending...
I20240501 13:59:09.637804 22704 ts_manager.cc:194] Registered new tserver with Master: 170e490f2fc44012a11802033eccc5f9 (127.22.33.65:33589)
I20240501 13:59:09.640221 22704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.65:47877
W20240501 13:59:09.960556 22895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:09.960887 22895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:09.961011 22895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:09.969012 22895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:09.969205 22895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.66
I20240501 13:59:09.978669 22895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.66
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:09.980136 22895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:09.981972 22895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:09.994853 22900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.995481 22901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.996390 22903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:09.996621 22895 server_base.cc:1034] running on GCE node
I20240501 13:59:09.997408 22895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:09.998461 22895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:09.999702 22895 hybrid_clock.cc:648] HybridClock initialized: now 1714571949999630 us; error 40 us; skew 500 ppm
I20240501 13:59:10.000080 22895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.002678 22895 webserver.cc:460] Webserver started at http://127.22.33.66:36351/ using document root <none> and password file <none>
I20240501 13:59:10.003474 22895 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.003609 22895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.003922 22895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.006736 22895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e4e527cc3b8141a7ab334df7fbbacb5d"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:10.007524 22895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e4e527cc3b8141a7ab334df7fbbacb5d"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:10.012913 22895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 13:59:10.016556 22909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.018059 22895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:10.018229 22895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e4e527cc3b8141a7ab334df7fbbacb5d"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-3610"
I20240501 13:59:10.018409 22895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.033998 22895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.034860 22895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:10.035151 22895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.036130 22895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.037690 22895 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.037762 22895 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.037886 22895 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.037937 22895 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.069340 22895 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.66:38315
I20240501 13:59:10.069373 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.66:38315 every 8 connection(s)
I20240501 13:59:10.070675 22895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:10.076117 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 22895
I20240501 13:59:10.076311 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:10.079777 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:0
--local_ip_for_outbound_sockets=127.22.33.67
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:10.085750 23022 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38743
I20240501 13:59:10.086081 23022 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.086907 23022 heartbeater.cc:511] Master 127.22.33.126:38743 requested a full tablet report, sending...
I20240501 13:59:10.088555 22704 ts_manager.cc:194] Registered new tserver with Master: e4e527cc3b8141a7ab334df7fbbacb5d (127.22.33.66:38315)
I20240501 13:59:10.089795 22704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.66:44775
W20240501 13:59:10.425906 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:10.426357 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:10.426594 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:10.436213 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:10.436470 23025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 13:59:10.446170 23025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:10.447708 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:10.449674 23025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:10.462399 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.463066 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.463722 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.464053 23025 server_base.cc:1034] running on GCE node
I20240501 13:59:10.464752 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:10.465938 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:10.467171 23025 hybrid_clock.cc:648] HybridClock initialized: now 1714571950467092 us; error 61 us; skew 500 ppm
I20240501 13:59:10.467491 23025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.470360 23025 webserver.cc:460] Webserver started at http://127.22.33.67:37671/ using document root <none> and password file <none>
I20240501 13:59:10.471935 23025 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.472252 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.473019 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.476888 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f5ef50941ff24e2a9cfa1e034fb80058"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.477814 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f5ef50941ff24e2a9cfa1e034fb80058"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.483736 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240501 13:59:10.487691 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.489203 23025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:10.489442 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f5ef50941ff24e2a9cfa1e034fb80058"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.489635 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.521936 23025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.522858 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:10.523155 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.524153 23025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.525636 23025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.525713 23025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.525849 23025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.525899 23025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.555898 23025 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:39321
I20240501 13:59:10.555948 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:39321 every 8 connection(s)
I20240501 13:59:10.557260 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:10.560482 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23025
I20240501 13:59:10.560659 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:10.563884 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:0
--local_ip_for_outbound_sockets=127.22.33.68
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:10.571952 23152 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38743
I20240501 13:59:10.572260 23152 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.573140 23152 heartbeater.cc:511] Master 127.22.33.126:38743 requested a full tablet report, sending...
I20240501 13:59:10.574697 22704 ts_manager.cc:194] Registered new tserver with Master: f5ef50941ff24e2a9cfa1e034fb80058 (127.22.33.67:39321)
I20240501 13:59:10.575486 22704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:58485
I20240501 13:59:10.643764 22891 heartbeater.cc:503] Master 127.22.33.126:38743 was elected leader, sending a full tablet report...
W20240501 13:59:10.898372 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:10.898692 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:10.898787 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:10.906524 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:10.906755 23155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 13:59:10.915971 23155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:10.917397 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:10.919193 23155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:10.932621 23161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.932785 23160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.933368 23163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.933943 23155 server_base.cc:1034] running on GCE node
I20240501 13:59:10.934649 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:10.935619 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:10.936851 23155 hybrid_clock.cc:648] HybridClock initialized: now 1714571950936768 us; error 53 us; skew 500 ppm
I20240501 13:59:10.937208 23155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.940290 23155 webserver.cc:460] Webserver started at http://127.22.33.68:35307/ using document root <none> and password file <none>
I20240501 13:59:10.941608 23155 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.941766 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.942137 23155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.945222 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.946134 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.952105 23155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240501 13:59:10.955881 23169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.957473 23155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:10.957653 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-3610"
I20240501 13:59:10.957852 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.984330 23155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.985282 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:10.985579 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.986569 23155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.988313 23155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.988390 23155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.988524 23155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.988576 23155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:11.021608 23155 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:38673
I20240501 13:59:11.021641 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:38673 every 8 connection(s)
I20240501 13:59:11.022958 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:11.032681 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23155
I20240501 13:59:11.032855 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:11.036646 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:0
--local_ip_for_outbound_sockets=127.22.33.69
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--builtin_ntp_servers=127.22.33.84:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:11.037336 23282 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38743
I20240501 13:59:11.037667 23282 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:11.038534 23282 heartbeater.cc:511] Master 127.22.33.126:38743 requested a full tablet report, sending...
I20240501 13:59:11.040012 22704 ts_manager.cc:194] Registered new tserver with Master: e45a53cfdcfc44988c3ed6ed61e5ccfc (127.22.33.68:38673)
I20240501 13:59:11.040866 22704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:45747
I20240501 13:59:11.092821 23022 heartbeater.cc:503] Master 127.22.33.126:38743 was elected leader, sending a full tablet report...
W20240501 13:59:11.378093 23285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:11.378412 23285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:11.378557 23285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:11.386349 23285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:11.386585 23285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 13:59:11.396179 23285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:11.397631 23285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:11.399394 23285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:11.411461 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:11.411967 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:11.412534 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:11.412853 23285 server_base.cc:1034] running on GCE node
I20240501 13:59:11.413504 23285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:11.414508 23285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:11.415733 23285 hybrid_clock.cc:648] HybridClock initialized: now 1714571951415660 us; error 57 us; skew 500 ppm
I20240501 13:59:11.416033 23285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:11.418336 23285 webserver.cc:460] Webserver started at http://127.22.33.69:41391/ using document root <none> and password file <none>
I20240501 13:59:11.419066 23285 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:11.419195 23285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:11.419520 23285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:11.422327 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c80c98b75e7c4be79e757c76012be9a4"
format_stamp: "Formatted at 2024-05-01 13:59:11 on dist-test-slave-3610"
I20240501 13:59:11.423120 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c80c98b75e7c4be79e757c76012be9a4"
format_stamp: "Formatted at 2024-05-01 13:59:11 on dist-test-slave-3610"
I20240501 13:59:11.428364 23285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:59:11.431797 23299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:11.433199 23285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 13:59:11.433364 23285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c80c98b75e7c4be79e757c76012be9a4"
format_stamp: "Formatted at 2024-05-01 13:59:11 on dist-test-slave-3610"
I20240501 13:59:11.433552 23285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:11.458432 23285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:11.459324 23285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:11.459625 23285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:11.460572 23285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:11.462229 23285 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:11.462309 23285 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:11.462438 23285 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:11.462492 23285 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:11.494185 23285 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:42655
I20240501 13:59:11.494241 23411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:42655 every 8 connection(s)
I20240501 13:59:11.495661 23285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:11.505786 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23285
I20240501 13:59:11.505961 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:11.508190 23412 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38743
I20240501 13:59:11.508497 23412 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:11.509352 23412 heartbeater.cc:511] Master 127.22.33.126:38743 requested a full tablet report, sending...
I20240501 13:59:11.510869 22703 ts_manager.cc:194] Registered new tserver with Master: c80c98b75e7c4be79e757c76012be9a4 (127.22.33.69:42655)
I20240501 13:59:11.511688 22703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:53235
I20240501 13:59:11.523047 22661 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:11.544636 22703 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43088:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:11.578195 23152 heartbeater.cc:503] Master 127.22.33.126:38743 was elected leader, sending a full tablet report...
I20240501 13:59:11.582618 23217 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd2db852eb4b25bfb368ca3a25833e (DEFAULT_TABLE table=TestTable [id=857ce190164943e78ea0d2ad1d999a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:11.584015 23217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd2db852eb4b25bfb368ca3a25833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:11.588238 22826 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd2db852eb4b25bfb368ca3a25833e (DEFAULT_TABLE table=TestTable [id=857ce190164943e78ea0d2ad1d999a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:11.589567 22826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd2db852eb4b25bfb368ca3a25833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:11.594348 23087 tablet_service.cc:1467] Processing CreateTablet for tablet 24bd2db852eb4b25bfb368ca3a25833e (DEFAULT_TABLE table=TestTable [id=857ce190164943e78ea0d2ad1d999a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:11.596154 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bd2db852eb4b25bfb368ca3a25833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:11.596416 23430 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Bootstrap starting.
I20240501 13:59:11.600387 23430 tablet_bootstrap.cc:654] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:11.601783 23430 log.cc:824] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:11.602507 23431 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Bootstrap starting.
I20240501 13:59:11.606293 23431 tablet_bootstrap.cc:654] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:11.607429 23430 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: No bootstrap required, opened a new log
I20240501 13:59:11.607729 23430 ts_tablet_manager.cc:1367] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 13:59:11.607715 23431 log.cc:824] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:11.608361 23433 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Bootstrap starting.
I20240501 13:59:11.611438 23431 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: No bootstrap required, opened a new log
I20240501 13:59:11.611708 23431 ts_tablet_manager.cc:1367] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 13:59:11.611997 23433 tablet_bootstrap.cc:654] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:11.613288 23433 log.cc:824] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:11.616230 23433 tablet_bootstrap.cc:492] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: No bootstrap required, opened a new log
I20240501 13:59:11.616525 23433 ts_tablet_manager.cc:1367] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 13:59:11.616652 23430 raft_consensus.cc:348] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.620155 23431 raft_consensus.cc:348] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.620635 23431 raft_consensus.cc:374] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:11.620715 23430 raft_consensus.cc:374] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:11.620821 23431 raft_consensus.cc:729] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e490f2fc44012a11802033eccc5f9, State: Initialized, Role: FOLLOWER
I20240501 13:59:11.620899 23430 raft_consensus.cc:729] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a53cfdcfc44988c3ed6ed61e5ccfc, State: Initialized, Role: FOLLOWER
I20240501 13:59:11.621678 23430 consensus_queue.cc:260] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.621814 23431 consensus_queue.cc:260] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.626767 23433 raft_consensus.cc:348] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.627311 23433 raft_consensus.cc:374] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:11.627372 23431 ts_tablet_manager.cc:1398] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240501 13:59:11.627472 23433 raft_consensus.cc:729] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ef50941ff24e2a9cfa1e034fb80058, State: Initialized, Role: FOLLOWER
I20240501 13:59:11.628454 23433 consensus_queue.cc:260] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.630520 23282 heartbeater.cc:503] Master 127.22.33.126:38743 was elected leader, sending a full tablet report...
I20240501 13:59:11.631033 23430 ts_tablet_manager.cc:1398] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Time spent starting tablet: real 0.023s user 0.014s sys 0.003s
I20240501 13:59:11.632797 23433 ts_tablet_manager.cc:1398] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20240501 13:59:11.775056 23283 tablet.cc:2369] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:11.809674 23153 tablet.cc:2369] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:11.872287 22892 tablet.cc:2369] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:11.937729 23437 raft_consensus.cc:482] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:11.938035 23437 raft_consensus.cc:504] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.939786 23437 leader_election.cc:288] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ef50941ff24e2a9cfa1e034fb80058 (127.22.33.67:39321), e45a53cfdcfc44988c3ed6ed61e5ccfc (127.22.33.68:38673)
I20240501 13:59:11.948213 23237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd2db852eb4b25bfb368ca3a25833e" candidate_uuid: "170e490f2fc44012a11802033eccc5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" is_pre_election: true
I20240501 13:59:11.948213 23107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd2db852eb4b25bfb368ca3a25833e" candidate_uuid: "170e490f2fc44012a11802033eccc5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" is_pre_election: true
I20240501 13:59:11.948822 23107 raft_consensus.cc:2454] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e490f2fc44012a11802033eccc5f9 in term 0.
I20240501 13:59:11.948822 23237 raft_consensus.cc:2454] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e490f2fc44012a11802033eccc5f9 in term 0.
I20240501 13:59:11.949584 22781 leader_election.cc:302] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e490f2fc44012a11802033eccc5f9, e45a53cfdcfc44988c3ed6ed61e5ccfc; no voters:
I20240501 13:59:11.950052 23437 raft_consensus.cc:2781] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:11.950294 23437 raft_consensus.cc:482] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:11.950392 23437 raft_consensus.cc:3037] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:11.952504 23437 raft_consensus.cc:504] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.953191 23437 leader_election.cc:288] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [CANDIDATE]: Term 1 election: Requested vote from peers f5ef50941ff24e2a9cfa1e034fb80058 (127.22.33.67:39321), e45a53cfdcfc44988c3ed6ed61e5ccfc (127.22.33.68:38673)
I20240501 13:59:11.953779 23237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd2db852eb4b25bfb368ca3a25833e" candidate_uuid: "170e490f2fc44012a11802033eccc5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc"
I20240501 13:59:11.953779 23107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bd2db852eb4b25bfb368ca3a25833e" candidate_uuid: "170e490f2fc44012a11802033eccc5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ef50941ff24e2a9cfa1e034fb80058"
I20240501 13:59:11.954015 23237 raft_consensus.cc:3037] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:11.954041 23107 raft_consensus.cc:3037] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:11.957314 23237 raft_consensus.cc:2454] T 24bd2db852eb4b25bfb368ca3a25833e P e45a53cfdcfc44988c3ed6ed61e5ccfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e490f2fc44012a11802033eccc5f9 in term 1.
I20240501 13:59:11.957314 23107 raft_consensus.cc:2454] T 24bd2db852eb4b25bfb368ca3a25833e P f5ef50941ff24e2a9cfa1e034fb80058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e490f2fc44012a11802033eccc5f9 in term 1.
I20240501 13:59:11.957890 22779 leader_election.cc:302] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e490f2fc44012a11802033eccc5f9, f5ef50941ff24e2a9cfa1e034fb80058; no voters:
I20240501 13:59:11.958276 23437 raft_consensus.cc:2781] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:11.958715 23437 raft_consensus.cc:686] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 170e490f2fc44012a11802033eccc5f9, State: Running, Role: LEADER
I20240501 13:59:11.959185 23437 consensus_queue.cc:237] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } }
I20240501 13:59:11.964280 22703 catalog_manager.cc:5484] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 170e490f2fc44012a11802033eccc5f9 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "170e490f2fc44012a11802033eccc5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e490f2fc44012a11802033eccc5f9" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 33589 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:12.028064 22661 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:12.031347 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 170e490f2fc44012a11802033eccc5f9 to finish bootstrapping
I20240501 13:59:12.047883 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e4e527cc3b8141a7ab334df7fbbacb5d to finish bootstrapping
I20240501 13:59:12.056535 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f5ef50941ff24e2a9cfa1e034fb80058 to finish bootstrapping
I20240501 13:59:12.064165 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e45a53cfdcfc44988c3ed6ed61e5ccfc to finish bootstrapping
I20240501 13:59:12.071969 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c80c98b75e7c4be79e757c76012be9a4 to finish bootstrapping
I20240501 13:59:12.514039 23412 heartbeater.cc:503] Master 127.22.33.126:38743 was elected leader, sending a full tablet report...
I20240501 13:59:12.553185 22826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:12.559309 23347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.562073 23217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.564373 23477 consensus_queue.cc:1035] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ef50941ff24e2a9cfa1e034fb80058" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:12.569548 23478 consensus_queue.cc:1035] T 24bd2db852eb4b25bfb368ca3a25833e P 170e490f2fc44012a11802033eccc5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e45a53cfdcfc44988c3ed6ed61e5ccfc" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:12.592154 23087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.597375 22957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.774966 22826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:12.789755 23347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.790256 23217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.791659 22957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.795257 23087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:12.950089 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 22764
I20240501 13:59:12.951607 22886 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.050840 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 22764
W20240501 13:59:13.075628 23041 connection.cc:535] server connection from 127.22.33.65:58573 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:13.076190 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 22895
I20240501 13:59:13.077270 23017 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.162343 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 22895
W20240501 13:59:13.180290 22688 connection.cc:535] server connection from 127.22.33.66:44775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:13.180541 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23025
I20240501 13:59:13.181517 23146 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.271268 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23025
I20240501 13:59:13.288875 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23155
I20240501 13:59:13.289937 23276 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.378788 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23155
I20240501 13:59:13.396600 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23285
I20240501 13:59:13.397697 23407 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.483023 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23285
I20240501 13:59:13.499960 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 22673
I20240501 13:59:13.501199 22734 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:13.609684 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 22673
2024-05-01T13:59:13Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (5095 ms)
[----------] 1 test from AdminCliTest (5095 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 13:59:13.652657 22661 test_util.cc:268] Using random seed: -465747061
I20240501 13:59:13.654525 22661 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:13.654608 22661 ts_itest-base.cc:113] --------------
I20240501 13:59:13.654661 22661 ts_itest-base.cc:114] 5 tablet servers
I20240501 13:59:13.654727 22661 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:13.654779 22661 ts_itest-base.cc:116] --------------
2024-05-01T13:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:13Z Disabled control of system clock
I20240501 13:59:13.678295 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:38943
--webserver_interface=127.22.33.126
--webserver_port=0
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.33.126:38943
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 13:59:14.012235 23535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.012555 23535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.012646 23535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.020172 23535 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:14.020308 23535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:14.020350 23535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.020408 23535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:14.020455 23535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:14.029785 23535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.33.126:38943
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.33.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.031177 23535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.032892 23535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.043785 23540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.044497 23541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.044936 23543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.045298 23535 server_base.cc:1034] running on GCE node
I20240501 13:59:14.046237 23535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.047977 23535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.049211 23535 hybrid_clock.cc:648] HybridClock initialized: now 1714571954049147 us; error 55 us; skew 500 ppm
I20240501 13:59:14.049546 23535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.052220 23535 webserver.cc:460] Webserver started at http://127.22.33.126:36783/ using document root <none> and password file <none>
I20240501 13:59:14.053364 23535 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.053503 23535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.053941 23535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.058889 23535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "484ab0c7bfd847acb2f6dbda183e121b"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.059873 23535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "484ab0c7bfd847acb2f6dbda183e121b"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.065337 23535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 13:59:14.068675 23549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.070098 23535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:14.070273 23535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
uuid: "484ab0c7bfd847acb2f6dbda183e121b"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.070473 23535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.105288 23535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.106237 23535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.106540 23535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.125715 23535 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.126:38943
I20240501 13:59:14.125748 23600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.126:38943 every 8 connection(s)
I20240501 13:59:14.127272 23535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:14.131556 23601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.137588 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23535
I20240501 13:59:14.137809 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:14.141650 23601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b: Bootstrap starting.
I20240501 13:59:14.144944 23601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.146584 23601 log.cc:824] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:14.149689 23601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b: No bootstrap required, opened a new log
I20240501 13:59:14.156569 23601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } }
I20240501 13:59:14.156919 23601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.157053 23601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484ab0c7bfd847acb2f6dbda183e121b, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.157668 23601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } }
I20240501 13:59:14.157860 23601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:14.157955 23601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:14.158087 23601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.160234 23601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } }
I20240501 13:59:14.160699 23601 leader_election.cc:302] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484ab0c7bfd847acb2f6dbda183e121b; no voters:
I20240501 13:59:14.161263 23601 leader_election.cc:288] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:14.161404 23606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.162575 23606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [term 1 LEADER]: Becoming Leader. State: Replica: 484ab0c7bfd847acb2f6dbda183e121b, State: Running, Role: LEADER
I20240501 13:59:14.163141 23606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } }
I20240501 13:59:14.163508 23601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:14.166038 23608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484ab0c7bfd847acb2f6dbda183e121b. Latest consensus state: current_term: 1 leader_uuid: "484ab0c7bfd847acb2f6dbda183e121b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } } }
I20240501 13:59:14.166355 23608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:14.166128 23607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484ab0c7bfd847acb2f6dbda183e121b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ab0c7bfd847acb2f6dbda183e121b" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 38943 } } }
I20240501 13:59:14.166464 23607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:14.168785 23613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:14.173796 23613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:14.182097 23613 catalog_manager.cc:1347] Generated new cluster ID: 0cc77beef57542ae92661c39925baad8
I20240501 13:59:14.182250 23613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:14.196774 23613 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:14.198556 23613 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:14.221654 23613 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 484ab0c7bfd847acb2f6dbda183e121b: Generated new TSK 0
I20240501 13:59:14.222860 23613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:14.236246 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.65:0
--local_ip_for_outbound_sockets=127.22.33.65
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 13:59:14.577354 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.577658 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.577781 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.585336 23625 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:14.585471 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.585633 23625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.65
I20240501 13:59:14.594875 23625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.65
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.596230 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.598034 23625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.609887 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.610445 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.611094 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.611400 23625 server_base.cc:1034] running on GCE node
I20240501 13:59:14.612016 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.613049 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.614259 23625 hybrid_clock.cc:648] HybridClock initialized: now 1714571954614203 us; error 56 us; skew 500 ppm
I20240501 13:59:14.614586 23625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.617033 23625 webserver.cc:460] Webserver started at http://127.22.33.65:39801/ using document root <none> and password file <none>
I20240501 13:59:14.617813 23625 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.617935 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.618275 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.620932 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c4c3a4e634f4d98b4e9d30456101daf"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.621783 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c4c3a4e634f4d98b4e9d30456101daf"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.627213 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:14.630870 23639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.632313 23625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:14.632484 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c4c3a4e634f4d98b4e9d30456101daf"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-3610"
I20240501 13:59:14.632695 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.649148 23625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.650024 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.650300 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.651293 23625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:14.652871 23625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:14.652988 23625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.653108 23625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:14.653182 23625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.685074 23625 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.65:42899
I20240501 13:59:14.685144 23751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.65:42899 every 8 connection(s)
I20240501 13:59:14.686424 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:14.694756 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23625
I20240501 13:59:14.694960 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:14.698422 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.66:0
--local_ip_for_outbound_sockets=127.22.33.66
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:14.699442 23752 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38943
I20240501 13:59:14.699750 23752 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:14.700520 23752 heartbeater.cc:511] Master 127.22.33.126:38943 requested a full tablet report, sending...
I20240501 13:59:14.702673 23566 ts_manager.cc:194] Registered new tserver with Master: 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65:42899)
I20240501 13:59:14.704190 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.65:57037
W20240501 13:59:15.029742 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.030063 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.030195 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.037885 23756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:15.038017 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.038182 23756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.66
I20240501 13:59:15.048063 23756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.66
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.049597 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.051466 23756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.064661 23761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.064780 23764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.065390 23762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.065943 23756 server_base.cc:1034] running on GCE node
I20240501 13:59:15.066612 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.067608 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.068835 23756 hybrid_clock.cc:648] HybridClock initialized: now 1714571955068760 us; error 69 us; skew 500 ppm
I20240501 13:59:15.069298 23756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.071727 23756 webserver.cc:460] Webserver started at http://127.22.33.66:43587/ using document root <none> and password file <none>
I20240501 13:59:15.072527 23756 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.072650 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.073014 23756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.075826 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c5f2a503ea7443ad84d5a0ceb53a2545"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.076617 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c5f2a503ea7443ad84d5a0ceb53a2545"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.082242 23756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20240501 13:59:15.086019 23770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.087543 23756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:15.087730 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c5f2a503ea7443ad84d5a0ceb53a2545"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.087955 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.101346 23756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.102428 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.102758 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.103893 23756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.105679 23756 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.105795 23756 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.105938 23756 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.106041 23756 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.141613 23756 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.66:33985
I20240501 13:59:15.141654 23882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.66:33985 every 8 connection(s)
I20240501 13:59:15.143157 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:15.146847 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23756
I20240501 13:59:15.147091 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:15.150259 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:0
--local_ip_for_outbound_sockets=127.22.33.67
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:15.158738 23883 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38943
I20240501 13:59:15.159139 23883 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.160192 23883 heartbeater.cc:511] Master 127.22.33.126:38943 requested a full tablet report, sending...
I20240501 13:59:15.161963 23566 ts_manager.cc:194] Registered new tserver with Master: c5f2a503ea7443ad84d5a0ceb53a2545 (127.22.33.66:33985)
I20240501 13:59:15.162793 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.66:40643
W20240501 13:59:15.494905 23886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.495234 23886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.495379 23886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.503177 23886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:15.503331 23886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.503515 23886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 13:59:15.513087 23886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.514526 23886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.516352 23886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.528756 23892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.529294 23891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.529996 23894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.530735 23886 server_base.cc:1034] running on GCE node
I20240501 13:59:15.531375 23886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.532371 23886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.533586 23886 hybrid_clock.cc:648] HybridClock initialized: now 1714571955533535 us; error 48 us; skew 500 ppm
I20240501 13:59:15.533922 23886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.536307 23886 webserver.cc:460] Webserver started at http://127.22.33.67:37355/ using document root <none> and password file <none>
I20240501 13:59:15.537148 23886 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.537268 23886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.537606 23886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.540638 23886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8d062f8f0c3148898ead71493c7b257c"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.541548 23886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8d062f8f0c3148898ead71493c7b257c"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.546815 23886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:15.550506 23900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.552021 23886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:15.552228 23886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8d062f8f0c3148898ead71493c7b257c"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:15.552465 23886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.570521 23886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.571451 23886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.571725 23886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.572701 23886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.574303 23886 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.574400 23886 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.574513 23886 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.574594 23886 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.606346 23886 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:34571
I20240501 13:59:15.606402 24012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:34571 every 8 connection(s)
I20240501 13:59:15.607690 23886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:15.610420 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 23886
I20240501 13:59:15.610592 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:15.613950 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:0
--local_ip_for_outbound_sockets=127.22.33.68
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:15.621577 24013 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38943
I20240501 13:59:15.621932 24013 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.622861 24013 heartbeater.cc:511] Master 127.22.33.126:38943 requested a full tablet report, sending...
I20240501 13:59:15.624497 23566 ts_manager.cc:194] Registered new tserver with Master: 8d062f8f0c3148898ead71493c7b257c (127.22.33.67:34571)
I20240501 13:59:15.625789 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:45949
I20240501 13:59:15.707039 23752 heartbeater.cc:503] Master 127.22.33.126:38943 was elected leader, sending a full tablet report...
W20240501 13:59:15.953576 24016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.953884 24016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.954046 24016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.962836 24016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:15.962968 24016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.963100 24016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 13:59:15.972791 24016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.974285 24016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.976276 24016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.989013 24022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.989591 24021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.990154 24024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.990535 24016 server_base.cc:1034] running on GCE node
I20240501 13:59:15.991171 24016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.992218 24016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.993438 24016 hybrid_clock.cc:648] HybridClock initialized: now 1714571955993376 us; error 54 us; skew 500 ppm
I20240501 13:59:15.993779 24016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.996207 24016 webserver.cc:460] Webserver started at http://127.22.33.68:35827/ using document root <none> and password file <none>
I20240501 13:59:15.997020 24016 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.997150 24016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.997498 24016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.000209 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "72738c6a28ff4d8b8b103227ba117d48"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:16.001070 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "72738c6a28ff4d8b8b103227ba117d48"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:16.006628 24016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:16.010171 24030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.011550 24016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:16.011711 24016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "72738c6a28ff4d8b8b103227ba117d48"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-3610"
I20240501 13:59:16.011919 24016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.029942 24016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.030881 24016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.031168 24016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.032140 24016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.033815 24016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.033916 24016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.034032 24016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.034103 24016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.066077 24016 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:46039
I20240501 13:59:16.066145 24142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:46039 every 8 connection(s)
I20240501 13:59:16.067443 24016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:16.074113 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24016
I20240501 13:59:16.074373 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:16.078049 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:0
--local_ip_for_outbound_sockets=127.22.33.69
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--builtin_ntp_servers=127.22.33.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:16.081758 24143 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38943
I20240501 13:59:16.082127 24143 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:16.082947 24143 heartbeater.cc:511] Master 127.22.33.126:38943 requested a full tablet report, sending...
I20240501 13:59:16.084488 23566 ts_manager.cc:194] Registered new tserver with Master: 72738c6a28ff4d8b8b103227ba117d48 (127.22.33.68:46039)
I20240501 13:59:16.085714 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:47571
I20240501 13:59:16.166477 23883 heartbeater.cc:503] Master 127.22.33.126:38943 was elected leader, sending a full tablet report...
W20240501 13:59:16.413892 24146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:16.414217 24146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:16.414305 24146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:16.421936 24146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:16.422107 24146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:16.422354 24146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 13:59:16.435149 24146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:38943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:16.436501 24146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:16.438336 24146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:16.450547 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.451097 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.451783 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:16.452523 24146 server_base.cc:1034] running on GCE node
I20240501 13:59:16.453305 24146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:16.454326 24146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:16.455549 24146 hybrid_clock.cc:648] HybridClock initialized: now 1714571956455488 us; error 55 us; skew 500 ppm
I20240501 13:59:16.455897 24146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:16.458429 24146 webserver.cc:460] Webserver started at http://127.22.33.69:41063/ using document root <none> and password file <none>
I20240501 13:59:16.459273 24146 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:16.459374 24146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:16.459683 24146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.462406 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ace61d0468fe4449b4fdbdfdea600def"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-3610"
I20240501 13:59:16.463160 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ace61d0468fe4449b4fdbdfdea600def"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-3610"
I20240501 13:59:16.468766 24146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 13:59:16.472358 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.473887 24146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:16.474081 24146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ace61d0468fe4449b4fdbdfdea600def"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-3610"
I20240501 13:59:16.474292 24146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.499789 24146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.500639 24146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.500881 24146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.501863 24146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.503309 24146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.503391 24146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.503472 24146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.503526 24146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.536183 24146 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:40681
I20240501 13:59:16.536227 24272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:40681 every 8 connection(s)
I20240501 13:59:16.537724 24146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:16.547858 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24146
I20240501 13:59:16.548038 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:16.550611 24273 heartbeater.cc:347] Connected to a master server at 127.22.33.126:38943
I20240501 13:59:16.550916 24273 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:16.551642 24273 heartbeater.cc:511] Master 127.22.33.126:38943 requested a full tablet report, sending...
I20240501 13:59:16.552911 23566 ts_manager.cc:194] Registered new tserver with Master: ace61d0468fe4449b4fdbdfdea600def (127.22.33.69:40681)
I20240501 13:59:16.553838 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:35369
I20240501 13:59:16.564385 22661 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:16.581038 23566 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44402:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:16.618604 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 998b230eb7a2455d9e78b4b0aedf75b9 (DEFAULT_TABLE table=TestTable [id=8a201b006246451d9464882ffd9a43dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:16.620714 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998b230eb7a2455d9e78b4b0aedf75b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:16.626796 24208 tablet_service.cc:1467] Processing CreateTablet for tablet 998b230eb7a2455d9e78b4b0aedf75b9 (DEFAULT_TABLE table=TestTable [id=8a201b006246451d9464882ffd9a43dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:16.628410 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998b230eb7a2455d9e78b4b0aedf75b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:16.629078 24013 heartbeater.cc:503] Master 127.22.33.126:38943 was elected leader, sending a full tablet report...
I20240501 13:59:16.629231 23948 tablet_service.cc:1467] Processing CreateTablet for tablet 998b230eb7a2455d9e78b4b0aedf75b9 (DEFAULT_TABLE table=TestTable [id=8a201b006246451d9464882ffd9a43dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:16.630597 23948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998b230eb7a2455d9e78b4b0aedf75b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:16.637773 24291 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Bootstrap starting.
I20240501 13:59:16.641852 24291 tablet_bootstrap.cc:654] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:16.642230 24292 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Bootstrap starting.
I20240501 13:59:16.643733 24291 log.cc:824] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:16.646576 24291 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: No bootstrap required, opened a new log
I20240501 13:59:16.646836 24291 ts_tablet_manager.cc:1367] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 13:59:16.648018 24292 tablet_bootstrap.cc:654] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:16.648588 24293 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Bootstrap starting.
I20240501 13:59:16.649617 24292 log.cc:824] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:16.652223 24293 tablet_bootstrap.cc:654] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:16.653056 24292 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: No bootstrap required, opened a new log
I20240501 13:59:16.653342 24293 log.cc:824] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:16.653364 24292 ts_tablet_manager.cc:1367] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 13:59:16.654793 24291 raft_consensus.cc:348] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.655267 24291 raft_consensus.cc:374] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:16.655375 24291 raft_consensus.cc:729] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4c3a4e634f4d98b4e9d30456101daf, State: Initialized, Role: FOLLOWER
I20240501 13:59:16.655803 24293 tablet_bootstrap.cc:492] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: No bootstrap required, opened a new log
I20240501 13:59:16.656091 24293 ts_tablet_manager.cc:1367] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 13:59:16.656126 24291 consensus_queue.cc:260] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.663226 24293 raft_consensus.cc:348] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.663674 24293 raft_consensus.cc:374] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:16.663820 24293 raft_consensus.cc:729] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace61d0468fe4449b4fdbdfdea600def, State: Initialized, Role: FOLLOWER
I20240501 13:59:16.663939 24291 ts_tablet_manager.cc:1398] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Time spent starting tablet: real 0.017s user 0.006s sys 0.010s
I20240501 13:59:16.664093 24292 raft_consensus.cc:348] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.664502 24292 raft_consensus.cc:374] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:16.664603 24292 raft_consensus.cc:729] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d062f8f0c3148898ead71493c7b257c, State: Initialized, Role: FOLLOWER
I20240501 13:59:16.664610 24293 consensus_queue.cc:260] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.665808 24273 heartbeater.cc:503] Master 127.22.33.126:38943 was elected leader, sending a full tablet report...
I20240501 13:59:16.666126 24292 consensus_queue.cc:260] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.666492 24293 ts_tablet_manager.cc:1398] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240501 13:59:16.669726 24292 ts_tablet_manager.cc:1398] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20240501 13:59:16.680375 24297 raft_consensus.cc:482] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:16.680640 24297 raft_consensus.cc:504] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.682456 24297 leader_election.cc:288] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ace61d0468fe4449b4fdbdfdea600def (127.22.33.69:40681), 8d062f8f0c3148898ead71493c7b257c (127.22.33.67:34571)
W20240501 13:59:16.689352 23753 tablet.cc:2369] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:16.690729 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace61d0468fe4449b4fdbdfdea600def" is_pre_election: true
I20240501 13:59:16.691304 24228 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4c3a4e634f4d98b4e9d30456101daf in term 0.
I20240501 13:59:16.692027 23641 leader_election.cc:302] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4c3a4e634f4d98b4e9d30456101daf, ace61d0468fe4449b4fdbdfdea600def; no voters:
I20240501 13:59:16.692330 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d062f8f0c3148898ead71493c7b257c" is_pre_election: true
I20240501 13:59:16.692471 24297 raft_consensus.cc:2781] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:16.692631 24297 raft_consensus.cc:482] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:16.692734 24297 raft_consensus.cc:3037] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:16.692842 23968 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4c3a4e634f4d98b4e9d30456101daf in term 0.
I20240501 13:59:16.695961 24297 raft_consensus.cc:504] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.696614 24297 leader_election.cc:288] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [CANDIDATE]: Term 1 election: Requested vote from peers ace61d0468fe4449b4fdbdfdea600def (127.22.33.69:40681), 8d062f8f0c3148898ead71493c7b257c (127.22.33.67:34571)
I20240501 13:59:16.697214 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace61d0468fe4449b4fdbdfdea600def"
I20240501 13:59:16.697162 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d062f8f0c3148898ead71493c7b257c"
I20240501 13:59:16.697496 23968 raft_consensus.cc:3037] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:16.697496 24228 raft_consensus.cc:3037] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:16.700631 23968 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4c3a4e634f4d98b4e9d30456101daf in term 1.
I20240501 13:59:16.700631 24228 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4c3a4e634f4d98b4e9d30456101daf in term 1.
I20240501 13:59:16.701078 23641 leader_election.cc:302] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4c3a4e634f4d98b4e9d30456101daf, ace61d0468fe4449b4fdbdfdea600def; no voters:
I20240501 13:59:16.701392 24297 raft_consensus.cc:2781] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:16.701817 24297 raft_consensus.cc:686] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [term 1 LEADER]: Becoming Leader. State: Replica: 2c4c3a4e634f4d98b4e9d30456101daf, State: Running, Role: LEADER
I20240501 13:59:16.702328 24297 consensus_queue.cc:237] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:16.706570 23565 catalog_manager.cc:5484] T 998b230eb7a2455d9e78b4b0aedf75b9 P 2c4c3a4e634f4d98b4e9d30456101daf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65). New cstate: current_term: 1 leader_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } } }
I20240501 13:59:16.732636 22661 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:16.734817 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2c4c3a4e634f4d98b4e9d30456101daf to finish bootstrapping
I20240501 13:59:16.743314 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c5f2a503ea7443ad84d5a0ceb53a2545 to finish bootstrapping
I20240501 13:59:16.751929 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8d062f8f0c3148898ead71493c7b257c to finish bootstrapping
I20240501 13:59:16.759667 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 72738c6a28ff4d8b8b103227ba117d48 to finish bootstrapping
I20240501 13:59:16.768257 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ace61d0468fe4449b4fdbdfdea600def to finish bootstrapping
I20240501 13:59:16.777230 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23625
I20240501 13:59:16.778692 23747 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 13:59:16.789355 24274 tablet.cc:2369] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:16.860210 24014 tablet.cc:2369] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:16.879751 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23625
I20240501 13:59:17.088408 24143 heartbeater.cc:503] Master 127.22.33.126:38943 was elected leader, sending a full tablet report...
I20240501 13:59:17.384918 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.385957 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.388854 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.390918 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.522867 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.527253 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.527860 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.530921 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.717417 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.723954 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.724289 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.736253 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.337829 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.344302 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.351776 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.352859 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.482600 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.489301 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.490849 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.491077 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.641530 24492 raft_consensus.cc:482] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:18.642319 24492 raft_consensus.cc:504] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:18.645499 24492 leader_election.cc:288] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65:42899), ace61d0468fe4449b4fdbdfdea600def (127.22.33.69:40681)
I20240501 13:59:18.660388 23948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:18.662735 23902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.65:42899: connect: Connection refused (error 111)
I20240501 13:59:18.676932 24078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:18.680420 23902 leader_election.cc:334] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65:42899): Network error: Client connection negotiation failed: client connection to 127.22.33.65:42899: connect: Connection refused (error 111)
I20240501 13:59:18.681291 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "8d062f8f0c3148898ead71493c7b257c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace61d0468fe4449b4fdbdfdea600def" is_pre_election: true
I20240501 13:59:18.681558 24228 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d062f8f0c3148898ead71493c7b257c in term 1.
I20240501 13:59:18.682183 23902 leader_election.cc:302] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d062f8f0c3148898ead71493c7b257c, ace61d0468fe4449b4fdbdfdea600def; no voters: 2c4c3a4e634f4d98b4e9d30456101daf
I20240501 13:59:18.682662 24492 raft_consensus.cc:2781] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:18.682785 24492 raft_consensus.cc:482] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:18.682865 24492 raft_consensus.cc:3037] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:18.686383 24492 raft_consensus.cc:504] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:18.687131 24492 leader_election.cc:288] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 election: Requested vote from peers 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65:42899), ace61d0468fe4449b4fdbdfdea600def (127.22.33.69:40681)
I20240501 13:59:18.688606 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998b230eb7a2455d9e78b4b0aedf75b9" candidate_uuid: "8d062f8f0c3148898ead71493c7b257c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace61d0468fe4449b4fdbdfdea600def"
I20240501 13:59:18.688843 24228 raft_consensus.cc:3037] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:18.691035 24228 raft_consensus.cc:2454] T 998b230eb7a2455d9e78b4b0aedf75b9 P ace61d0468fe4449b4fdbdfdea600def [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d062f8f0c3148898ead71493c7b257c in term 2.
I20240501 13:59:18.691702 23902 leader_election.cc:302] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d062f8f0c3148898ead71493c7b257c, ace61d0468fe4449b4fdbdfdea600def; no voters:
I20240501 13:59:18.692121 24492 raft_consensus.cc:2781] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:18.693040 24492 raft_consensus.cc:686] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [term 2 LEADER]: Becoming Leader. State: Replica: 8d062f8f0c3148898ead71493c7b257c, State: Running, Role: LEADER
I20240501 13:59:18.693753 24492 consensus_queue.cc:237] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } }
I20240501 13:59:18.697038 24208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.697072 23818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:18.699018 23902 leader_election.cc:334] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65:42899): Network error: Client connection negotiation failed: client connection to 127.22.33.65:42899: connect: Connection refused (error 111)
I20240501 13:59:18.708662 23566 catalog_manager.cc:5484] T 998b230eb7a2455d9e78b4b0aedf75b9 P 8d062f8f0c3148898ead71493c7b257c reported cstate change: term changed from 1 to 2, leader changed from 2c4c3a4e634f4d98b4e9d30456101daf (127.22.33.65) to 8d062f8f0c3148898ead71493c7b257c (127.22.33.67). New cstate: current_term: 2 leader_uuid: "8d062f8f0c3148898ead71493c7b257c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4c3a4e634f4d98b4e9d30456101daf" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 42899 } } peers { permanent_uuid: "ace61d0468fe4449b4fdbdfdea600def" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 40681 } } peers { permanent_uuid: "8d062f8f0c3148898ead71493c7b257c" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 34571 } } }
I20240501 13:59:18.844619 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23756
I20240501 13:59:18.846050 23877 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:18.937351 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23756
I20240501 13:59:18.954886 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23886
I20240501 13:59:18.955879 24008 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:19.056811 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23886
I20240501 13:59:19.080319 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24016
I20240501 13:59:19.081418 24137 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:19.173435 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24016
W20240501 13:59:19.191215 23551 connection.cc:535] server connection from 127.22.33.68:47571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:19.191493 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24146
I20240501 13:59:19.192476 24267 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:19.290004 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24146
I20240501 13:59:19.307924 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 23535
I20240501 13:59:19.309132 23596 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:19.424517 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 23535
2024-05-01T13:59:19Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5814 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5814 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 13:59:19.467545 22661 test_util.cc:268] Using random seed: -459932158
I20240501 13:59:19.469494 22661 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:19.469565 22661 ts_itest-base.cc:113] --------------
I20240501 13:59:19.469627 22661 ts_itest-base.cc:114] 7 tablet servers
I20240501 13:59:19.469676 22661 ts_itest-base.cc:115] 1 replicas per TS
I20240501 13:59:19.469748 22661 ts_itest-base.cc:116] --------------
2024-05-01T13:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:19Z Disabled control of system clock
I20240501 13:59:19.492666 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:41135
--webserver_interface=127.22.33.126
--webserver_port=0
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.33.126:41135
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 13:59:19.808434 24517 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 13:59:19.808748 24517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:19.808812 24517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:19.808912 24517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:19.816241 24517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:19.816361 24517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:19.816438 24517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:19.816476 24517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:19.816533 24517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:19.825755 24517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.33.126:41135
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.33.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:19.827145 24517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:19.828882 24517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:19.839380 24522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:19.839972 24525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:19.840655 24523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:19.841398 24517 server_base.cc:1034] running on GCE node
I20240501 13:59:19.842110 24517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:19.843432 24517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:19.844632 24517 hybrid_clock.cc:648] HybridClock initialized: now 1714571959844577 us; error 34 us; skew 500 ppm
I20240501 13:59:19.844914 24517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:19.847100 24517 webserver.cc:460] Webserver started at http://127.22.33.126:39471/ using document root <none> and password file <none>
I20240501 13:59:19.847798 24517 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:19.847903 24517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:19.848166 24517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:19.850745 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ad3ae472198240bf908bfe32012c5a39"
format_stamp: "Formatted at 2024-05-01 13:59:19 on dist-test-slave-3610"
I20240501 13:59:19.851437 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ad3ae472198240bf908bfe32012c5a39"
format_stamp: "Formatted at 2024-05-01 13:59:19 on dist-test-slave-3610"
I20240501 13:59:19.856563 24517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:19.860100 24531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:19.861523 24517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:19.861667 24517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ad3ae472198240bf908bfe32012c5a39"
format_stamp: "Formatted at 2024-05-01 13:59:19 on dist-test-slave-3610"
I20240501 13:59:19.861831 24517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:19.896561 24517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:19.897477 24517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:19.897742 24517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:19.915859 24517 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.126:41135
I20240501 13:59:19.915887 24582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.126:41135 every 8 connection(s)
I20240501 13:59:19.917317 24517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:19.918807 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24517
I20240501 13:59:19.919000 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:19.921897 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:19.932950 24583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39: Bootstrap starting.
I20240501 13:59:19.936116 24583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:19.937220 24583 log.cc:824] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:19.939955 24583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39: No bootstrap required, opened a new log
I20240501 13:59:19.946349 24583 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } }
I20240501 13:59:19.946722 24583 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:19.946799 24583 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3ae472198240bf908bfe32012c5a39, State: Initialized, Role: FOLLOWER
I20240501 13:59:19.947486 24583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } }
I20240501 13:59:19.947683 24583 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:19.947811 24583 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:19.947955 24583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:19.950191 24583 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } }
I20240501 13:59:19.950774 24583 leader_election.cc:302] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3ae472198240bf908bfe32012c5a39; no voters:
I20240501 13:59:19.951367 24583 leader_election.cc:288] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:19.951500 24588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:19.952105 24588 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [term 1 LEADER]: Becoming Leader. State: Replica: ad3ae472198240bf908bfe32012c5a39, State: Running, Role: LEADER
I20240501 13:59:19.952687 24588 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } }
I20240501 13:59:19.953375 24583 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:19.955511 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad3ae472198240bf908bfe32012c5a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } } }
I20240501 13:59:19.955904 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:19.957046 24595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:19.957005 24590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3ae472198240bf908bfe32012c5a39. Latest consensus state: current_term: 1 leader_uuid: "ad3ae472198240bf908bfe32012c5a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3ae472198240bf908bfe32012c5a39" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 41135 } } }
I20240501 13:59:19.957381 24590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:19.963084 24595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:19.972014 24595 catalog_manager.cc:1347] Generated new cluster ID: 02cc1e2e1d5e48c387cf3fe638900c82
I20240501 13:59:19.972275 24595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:19.989831 24595 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:19.991087 24595 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:19.999455 24595 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ad3ae472198240bf908bfe32012c5a39: Generated new TSK 0
I20240501 13:59:20.000550 24595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:20.011713 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.65:0
--local_ip_for_outbound_sockets=127.22.33.65
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 13:59:20.346486 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:20.346781 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:20.346920 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:20.355064 24607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:20.355201 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:20.355387 24607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.65
I20240501 13:59:20.365084 24607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.65
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:20.366509 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:20.368371 24607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.380640 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.380663 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.381359 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.382064 24607 server_base.cc:1034] running on GCE node
I20240501 13:59:20.382694 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:20.383698 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:20.384912 24607 hybrid_clock.cc:648] HybridClock initialized: now 1714571960384868 us; error 39 us; skew 500 ppm
I20240501 13:59:20.385277 24607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.387616 24607 webserver.cc:460] Webserver started at http://127.22.33.65:39527/ using document root <none> and password file <none>
I20240501 13:59:20.388435 24607 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.388556 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.388880 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.391721 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0e31c1e6d8f843558a29d08245b4c93d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.392526 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0e31c1e6d8f843558a29d08245b4c93d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.397739 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:20.401393 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.402729 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:20.402887 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0e31c1e6d8f843558a29d08245b4c93d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.403090 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.426839 24607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.428021 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:20.428373 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.429364 24607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:20.430826 24607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:20.430946 24607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.431092 24607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:20.431183 24607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.460932 24607 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.65:36237
I20240501 13:59:20.461057 24733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.65:36237 every 8 connection(s)
I20240501 13:59:20.462302 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:20.469957 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24607
I20240501 13:59:20.470280 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:20.473920 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.66:0
--local_ip_for_outbound_sockets=127.22.33.66
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:20.476186 24734 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:20.476491 24734 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.477255 24734 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:20.479138 24548 ts_manager.cc:194] Registered new tserver with Master: 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:20.480624 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.65:42555
W20240501 13:59:20.799105 24738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:20.799405 24738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:20.799521 24738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:20.807058 24738 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:20.807178 24738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:20.807314 24738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.66
I20240501 13:59:20.816188 24738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.66
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:20.817618 24738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:20.819371 24738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.831104 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.831692 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.832473 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.833076 24738 server_base.cc:1034] running on GCE node
I20240501 13:59:20.833621 24738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:20.834517 24738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:20.835698 24738 hybrid_clock.cc:648] HybridClock initialized: now 1714571960835646 us; error 43 us; skew 500 ppm
I20240501 13:59:20.835995 24738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.838199 24738 webserver.cc:460] Webserver started at http://127.22.33.66:41169/ using document root <none> and password file <none>
I20240501 13:59:20.838901 24738 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.839008 24738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.839315 24738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.841933 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "85c2edb136794ec7a6e4352c57090d79"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.842685 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "85c2edb136794ec7a6e4352c57090d79"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.847764 24738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 13:59:20.851045 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.852331 24738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:20.852478 24738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "85c2edb136794ec7a6e4352c57090d79"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-3610"
I20240501 13:59:20.852664 24738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.870285 24738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.871115 24738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:20.871378 24738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.872285 24738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:20.873744 24738 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:20.873828 24738 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.873945 24738 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:20.874020 24738 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.906595 24738 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.66:40637
I20240501 13:59:20.906648 24864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.66:40637 every 8 connection(s)
I20240501 13:59:20.907955 24738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:20.911659 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24738
I20240501 13:59:20.911845 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:20.914830 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:0
--local_ip_for_outbound_sockets=127.22.33.67
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:20.923166 24865 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:20.923493 24865 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.924228 24865 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:20.925585 24548 ts_manager.cc:194] Registered new tserver with Master: 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:20.926388 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.66:57337
W20240501 13:59:21.253250 24868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:21.253571 24868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:21.253692 24868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:21.261353 24868 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:21.261485 24868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:21.261634 24868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 13:59:21.271289 24868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:21.272680 24868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:21.274489 24868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:21.286703 24873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:21.287283 24874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:21.287788 24876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:21.288429 24868 server_base.cc:1034] running on GCE node
I20240501 13:59:21.289052 24868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:21.289985 24868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:21.291186 24868 hybrid_clock.cc:648] HybridClock initialized: now 1714571961291141 us; error 50 us; skew 500 ppm
I20240501 13:59:21.291515 24868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:21.293851 24868 webserver.cc:460] Webserver started at http://127.22.33.67:36717/ using document root <none> and password file <none>
I20240501 13:59:21.294598 24868 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:21.294706 24868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:21.295024 24868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:21.297660 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b481b70e6d1f41408327ab1de26d7406"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.298445 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b481b70e6d1f41408327ab1de26d7406"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.303639 24868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240501 13:59:21.307241 24882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.308656 24868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:21.308799 24868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b481b70e6d1f41408327ab1de26d7406"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.309041 24868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:21.321111 24868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:21.321966 24868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:21.322233 24868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:21.323184 24868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:21.324653 24868 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:21.324739 24868 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.324854 24868 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:21.324926 24868 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.356590 24868 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:37985
I20240501 13:59:21.356648 24994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:37985 every 8 connection(s)
I20240501 13:59:21.358070 24868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:21.366493 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24868
I20240501 13:59:21.366689 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:21.370600 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:0
--local_ip_for_outbound_sockets=127.22.33.68
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:21.372022 24995 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:21.372390 24995 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:21.373399 24995 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:21.374815 24548 ts_manager.cc:194] Registered new tserver with Master: b481b70e6d1f41408327ab1de26d7406 (127.22.33.67:37985)
I20240501 13:59:21.375589 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:37975
I20240501 13:59:21.483435 24734 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:21.700160 24998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:21.700502 24998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:21.700649 24998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:21.708289 24998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:21.708451 24998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:21.708619 24998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 13:59:21.718102 24998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:21.719501 24998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:21.721311 24998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:21.733605 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:21.734103 25006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:21.734727 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:21.735069 24998 server_base.cc:1034] running on GCE node
I20240501 13:59:21.735694 24998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:21.736708 24998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:21.737912 24998 hybrid_clock.cc:648] HybridClock initialized: now 1714571961737833 us; error 71 us; skew 500 ppm
I20240501 13:59:21.738241 24998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:21.740610 24998 webserver.cc:460] Webserver started at http://127.22.33.68:38397/ using document root <none> and password file <none>
I20240501 13:59:21.741431 24998 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:21.741533 24998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:21.741854 24998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:21.744501 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "51db4831ba4348c3a502b54ae65eed48"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.745329 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "51db4831ba4348c3a502b54ae65eed48"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.750562 24998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 13:59:21.754431 25012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.756016 24998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:21.756196 24998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "51db4831ba4348c3a502b54ae65eed48"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:21.756402 24998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:21.777671 24998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:21.778581 24998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:21.778851 24998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:21.779819 24998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:21.781350 24998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:21.781457 24998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.781579 24998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:21.781651 24998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:21.813757 24998 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:44081
I20240501 13:59:21.813759 25124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:44081 every 8 connection(s)
I20240501 13:59:21.815131 24998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:21.819577 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 24998
I20240501 13:59:21.819774 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:21.823241 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:0
--local_ip_for_outbound_sockets=127.22.33.69
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:21.829427 25125 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:21.829737 25125 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:21.830485 25125 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:21.831828 24548 ts_manager.cc:194] Registered new tserver with Master: 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68:44081)
I20240501 13:59:21.832628 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:59059
I20240501 13:59:21.929004 24865 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:22.156420 25128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:22.156734 25128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:22.156826 25128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:22.164515 25128 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:22.164644 25128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:22.164768 25128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 13:59:22.174101 25128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:22.175487 25128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:22.177328 25128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:22.188676 25134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.189177 25133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.189920 25136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:22.190613 25128 server_base.cc:1034] running on GCE node
I20240501 13:59:22.191246 25128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:22.192191 25128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:22.193377 25128 hybrid_clock.cc:648] HybridClock initialized: now 1714571962193312 us; error 61 us; skew 500 ppm
I20240501 13:59:22.193686 25128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:22.195956 25128 webserver.cc:460] Webserver started at http://127.22.33.69:45703/ using document root <none> and password file <none>
I20240501 13:59:22.196709 25128 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:22.196820 25128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:22.197185 25128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:22.199808 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ae9b30f780af481cbacd0566ab93e248"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.200583 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ae9b30f780af481cbacd0566ab93e248"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.205894 25128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:22.209584 25142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.211004 25128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:22.211230 25128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ae9b30f780af481cbacd0566ab93e248"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.211427 25128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:22.226663 25128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:22.227579 25128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:22.227852 25128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:22.228833 25128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:22.230444 25128 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:22.230530 25128 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.230610 25128 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:22.230664 25128 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.263257 25128 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:34503
I20240501 13:59:22.263309 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:34503 every 8 connection(s)
I20240501 13:59:22.264636 25128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:22.272491 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 25128
I20240501 13:59:22.272712 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:22.275818 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:0
--local_ip_for_outbound_sockets=127.22.33.70
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:22.283289 25255 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:22.283632 25255 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:22.284421 25255 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:22.285890 24548 ts_manager.cc:194] Registered new tserver with Master: ae9b30f780af481cbacd0566ab93e248 (127.22.33.69:34503)
I20240501 13:59:22.286644 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:42621
I20240501 13:59:22.378154 24995 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:22.616943 25258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:22.617313 25258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:22.617466 25258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:22.625119 25258 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:22.625275 25258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:22.625453 25258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 13:59:22.635190 25258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:22.636608 25258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:22.638409 25258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:22.650254 25263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.650813 25264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.651500 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:22.652112 25258 server_base.cc:1034] running on GCE node
I20240501 13:59:22.652706 25258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:22.653718 25258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:22.654946 25258 hybrid_clock.cc:648] HybridClock initialized: now 1714571962654879 us; error 54 us; skew 500 ppm
I20240501 13:59:22.655283 25258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:22.657871 25258 webserver.cc:460] Webserver started at http://127.22.33.70:35527/ using document root <none> and password file <none>
I20240501 13:59:22.658589 25258 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:22.658684 25258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:22.658960 25258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:22.661787 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "35f855d8b43040f7843747f312c33239"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.662609 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "35f855d8b43040f7843747f312c33239"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.667804 25258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:59:22.671140 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.672502 25258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:22.672629 25258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "35f855d8b43040f7843747f312c33239"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:22.672798 25258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:22.726930 25258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:22.727792 25258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:22.728035 25258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:22.728942 25258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:22.730417 25258 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:22.730515 25258 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.730641 25258 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:22.730715 25258 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.762194 25258 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:37695
I20240501 13:59:22.762251 25384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:37695 every 8 connection(s)
I20240501 13:59:22.763535 25258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:22.765671 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 25258
I20240501 13:59:22.765817 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 13:59:22.769400 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:0
--local_ip_for_outbound_sockets=127.22.33.71
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:22.777740 25385 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:22.778087 25385 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:22.778990 25385 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:22.780453 24548 ts_manager.cc:194] Registered new tserver with Master: 35f855d8b43040f7843747f312c33239 (127.22.33.70:37695)
I20240501 13:59:22.781302 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:35467
I20240501 13:59:22.835114 25125 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:23.108968 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:23.109366 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:23.109513 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:23.117480 25388 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:23.117655 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:23.117817 25388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 13:59:23.127216 25388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:23.128679 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:23.130486 25388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:23.143267 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.143327 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.143949 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:23.144567 25388 server_base.cc:1034] running on GCE node
I20240501 13:59:23.145232 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:23.146281 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:23.147517 25388 hybrid_clock.cc:648] HybridClock initialized: now 1714571963147465 us; error 48 us; skew 500 ppm
I20240501 13:59:23.147852 25388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:23.150211 25388 webserver.cc:460] Webserver started at http://127.22.33.71:39935/ using document root <none> and password file <none>
I20240501 13:59:23.150978 25388 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:23.151101 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:23.151423 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:23.154273 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2c3d9ee3986341868e534aef0a98e8b0"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-3610"
I20240501 13:59:23.155133 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2c3d9ee3986341868e534aef0a98e8b0"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-3610"
I20240501 13:59:23.160686 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 13:59:23.164347 25402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.165819 25388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:23.165987 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2c3d9ee3986341868e534aef0a98e8b0"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-3610"
I20240501 13:59:23.166222 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:23.190224 25388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:23.191090 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:23.191370 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:23.192346 25388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:23.193905 25388 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:23.193990 25388 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.194103 25388 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:23.194180 25388 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.226508 25388 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:37917
I20240501 13:59:23.226557 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:37917 every 8 connection(s)
I20240501 13:59:23.227855 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:23.229667 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 25388
I20240501 13:59:23.229847 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 13:59:23.241550 25515 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:23.241873 25515 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:23.242647 25515 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:23.244022 24548 ts_manager.cc:194] Registered new tserver with Master: 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71:37917)
I20240501 13:59:23.245061 24548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:58511
I20240501 13:59:23.246484 22661 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:23.263761 24548 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:23.289181 25255 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:23.294281 24930 tablet_service.cc:1467] Processing CreateTablet for tablet 5e87402f3d5d42e5a5df4cccc16a20b2 (DEFAULT_TABLE table=TestTable [id=1dd534206a024329a22498422bcb18f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.295629 24930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87402f3d5d42e5a5df4cccc16a20b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.305644 25529 tablet_bootstrap.cc:492] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap starting.
I20240501 13:59:23.308940 25529 tablet_bootstrap.cc:654] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.310011 25529 log.cc:824] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:23.312528 25529 tablet_bootstrap.cc:492] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: No bootstrap required, opened a new log
I20240501 13:59:23.312803 25529 ts_tablet_manager.cc:1367] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 13:59:23.319787 25529 raft_consensus.cc:348] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:23.320189 25529 raft_consensus.cc:374] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.320299 25529 raft_consensus.cc:729] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.321019 25529 consensus_queue.cc:260] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:23.321244 25529 raft_consensus.cc:388] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:23.321375 25529 raft_consensus.cc:482] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:23.321527 25529 raft_consensus.cc:3037] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.323630 25529 raft_consensus.cc:504] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:23.324152 25529 leader_election.cc:302] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b481b70e6d1f41408327ab1de26d7406; no voters:
I20240501 13:59:23.324826 25529 leader_election.cc:288] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:23.324944 25531 raft_consensus.cc:2781] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.325670 25531 raft_consensus.cc:686] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 LEADER]: Becoming Leader. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Running, Role: LEADER
I20240501 13:59:23.326234 25531 consensus_queue.cc:237] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:23.326655 25529 ts_tablet_manager.cc:1398] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20240501 13:59:23.331216 24548 catalog_manager.cc:5484] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 reported cstate change: term changed from 0 to 1, leader changed from <none> to b481b70e6d1f41408327ab1de26d7406 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "b481b70e6d1f41408327ab1de26d7406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:23.341037 22661 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:23.343317 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e31c1e6d8f843558a29d08245b4c93d to finish bootstrapping
I20240501 13:59:23.352967 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 85c2edb136794ec7a6e4352c57090d79 to finish bootstrapping
I20240501 13:59:23.363719 22661 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b481b70e6d1f41408327ab1de26d7406 to finish bootstrapping
I20240501 13:59:23.372712 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 51db4831ba4348c3a502b54ae65eed48 to finish bootstrapping
I20240501 13:59:23.381983 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ae9b30f780af481cbacd0566ab93e248 to finish bootstrapping
I20240501 13:59:23.391095 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 35f855d8b43040f7843747f312c33239 to finish bootstrapping
I20240501 13:59:23.400048 22661 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2c3d9ee3986341868e534aef0a98e8b0 to finish bootstrapping
I20240501 13:59:23.409838 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24868
I20240501 13:59:23.410873 24990 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:23.501314 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24868
I20240501 13:59:23.518118 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24998
I20240501 13:59:23.519292 25120 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:23.600694 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24998
I20240501 13:59:23.617108 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 25128
I20240501 13:59:23.618229 25250 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:23.699630 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 25128
I20240501 13:59:23.715876 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 25258
I20240501 13:59:23.717015 25380 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:23.797708 25385 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:23.801396 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 25258
I20240501 13:59:23.814481 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 25388
I20240501 13:59:23.815603 25510 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:23.896427 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 25388
I20240501 13:59:27.665973 24547 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:27.687903 24669 tablet_service.cc:1467] Processing CreateTablet for tablet c8caad758e6243e48008fe469425fcff (DEFAULT_TABLE table=rebalance_test_table_0 [id=2435645932f448f5a82db7bdb4911b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.687893 24800 tablet_service.cc:1467] Processing CreateTablet for tablet 04a8ca79aa05482c898141739d0ebd7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=2435645932f448f5a82db7bdb4911b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.687932 24668 tablet_service.cc:1467] Processing CreateTablet for tablet bb1d595a67cb40e195e9db6841978c0d (DEFAULT_TABLE table=rebalance_test_table_0 [id=2435645932f448f5a82db7bdb4911b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.689527 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a8ca79aa05482c898141739d0ebd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.689519 24668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1d595a67cb40e195e9db6841978c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.690032 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8caad758e6243e48008fe469425fcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.703621 25556 tablet_bootstrap.cc:492] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:27.704025 25557 tablet_bootstrap.cc:492] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:27.707773 25556 tablet_bootstrap.cc:654] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.707773 25557 tablet_bootstrap.cc:654] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.709177 25557 log.cc:824] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:27.709177 25556 log.cc:824] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:27.712280 25556 tablet_bootstrap.cc:492] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:27.712389 25557 tablet_bootstrap.cc:492] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:27.712600 25556 ts_tablet_manager.cc:1367] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:59:27.712611 25557 ts_tablet_manager.cc:1367] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:59:27.721984 25557 raft_consensus.cc:348] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:27.721984 25556 raft_consensus.cc:348] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.722460 25556 raft_consensus.cc:374] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.722502 25557 raft_consensus.cc:374] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.722620 25556 raft_consensus.cc:729] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.722640 25557 raft_consensus.cc:729] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.723450 25556 consensus_queue.cc:260] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.723444 25557 consensus_queue.cc:260] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:27.723712 25556 raft_consensus.cc:388] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:27.723713 25557 raft_consensus.cc:388] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:27.723856 25557 raft_consensus.cc:482] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:27.723856 25556 raft_consensus.cc:482] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:27.724032 25556 raft_consensus.cc:3037] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.724032 25557 raft_consensus.cc:3037] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.794572 25557 raft_consensus.cc:504] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:27.794596 25556 raft_consensus.cc:504] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.795267 25557 leader_election.cc:302] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:27.795274 25556 leader_election.cc:302] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:27.797662 25561 raft_consensus.cc:2781] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:27.797974 25557 leader_election.cc:288] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:27.798069 25560 raft_consensus.cc:2781] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:27.798455 25561 raft_consensus.cc:686] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:27.799122 25561 consensus_queue.cc:237] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.800374 25556 leader_election.cc:288] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:27.801155 25557 ts_tablet_manager.cc:1398] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.088s user 0.018s sys 0.004s
I20240501 13:59:27.801880 25560 raft_consensus.cc:686] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:27.801963 25556 ts_tablet_manager.cc:1398] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.089s user 0.014s sys 0.007s
I20240501 13:59:27.802438 25556 tablet_bootstrap.cc:492] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:27.803807 25556 tablet_bootstrap.cc:654] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.805426 25560 consensus_queue.cc:237] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:27.805603 24547 catalog_manager.cc:5484] T c8caad758e6243e48008fe469425fcff P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
I20240501 13:59:27.806478 25556 tablet_bootstrap.cc:492] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:27.806663 25556 ts_tablet_manager.cc:1367] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 13:59:27.807731 25556 raft_consensus.cc:348] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.808544 25556 raft_consensus.cc:374] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.808701 25556 raft_consensus.cc:729] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.809010 25556 consensus_queue.cc:260] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.809288 25556 raft_consensus.cc:388] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:27.809417 25556 raft_consensus.cc:482] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:27.809587 25556 raft_consensus.cc:3037] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.810983 24547 catalog_manager.cc:5484] T 04a8ca79aa05482c898141739d0ebd7d P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
I20240501 13:59:27.898095 25556 raft_consensus.cc:504] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.898504 25556 leader_election.cc:302] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:27.898739 25556 leader_election.cc:288] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:27.898902 25561 raft_consensus.cc:2781] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:27.899296 25561 raft_consensus.cc:686] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:27.899405 25556 ts_tablet_manager.cc:1398] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.093s user 0.001s sys 0.004s
I20240501 13:59:27.899580 25561 consensus_queue.cc:237] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:27.901903 24547 catalog_manager.cc:5484] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
W20240501 13:59:28.270510 25571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.270845 25571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.278352 25571 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 140.125 ms
time per row: 0.0700626 ms
I20240501 13:59:28.569669 24548 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:28.577751 24800 tablet_service.cc:1467] Processing CreateTablet for tablet 333670a3a08c4ef4b84def2560eda9c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3aa4500548d34d888c0efe6d86f95330]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.577844 24799 tablet_service.cc:1467] Processing CreateTablet for tablet 9f89ef680f644d51baf2fd632a429af7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3aa4500548d34d888c0efe6d86f95330]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.577980 24669 tablet_service.cc:1467] Processing CreateTablet for tablet b35dc4ce0bf144d19068653fdfd1e1b8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3aa4500548d34d888c0efe6d86f95330]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.578382 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f89ef680f644d51baf2fd632a429af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.578399 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35dc4ce0bf144d19068653fdfd1e1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.578629 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333670a3a08c4ef4b84def2560eda9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.584828 25615 tablet_bootstrap.cc:492] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:28.585543 25616 tablet_bootstrap.cc:492] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:28.586869 25615 tablet_bootstrap.cc:654] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.587445 25616 tablet_bootstrap.cc:654] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.589087 25615 tablet_bootstrap.cc:492] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:28.589231 25615 ts_tablet_manager.cc:1367] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 13:59:28.589907 25615 raft_consensus.cc:348] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:28.590068 25615 raft_consensus.cc:374] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.590165 25615 raft_consensus.cc:729] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.590406 25615 consensus_queue.cc:260] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:28.590483 25616 tablet_bootstrap.cc:492] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:28.590569 25615 raft_consensus.cc:388] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:28.590600 25616 ts_tablet_manager.cc:1367] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 13:59:28.590647 25615 raft_consensus.cc:482] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:28.590754 25615 raft_consensus.cc:3037] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.591435 25616 raft_consensus.cc:348] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.591583 25616 raft_consensus.cc:374] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.591715 25616 raft_consensus.cc:729] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.591938 25616 consensus_queue.cc:260] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.592125 25616 raft_consensus.cc:388] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:28.592206 25616 raft_consensus.cc:482] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:28.592315 25616 raft_consensus.cc:3037] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.630581 25616 raft_consensus.cc:504] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.630579 25615 raft_consensus.cc:504] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:28.630939 25615 leader_election.cc:302] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:28.630939 25616 leader_election.cc:302] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:28.631151 25615 leader_election.cc:288] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:28.631160 25616 leader_election.cc:288] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:28.631283 25606 raft_consensus.cc:2781] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.631485 25602 raft_consensus.cc:2781] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.631825 25615 ts_tablet_manager.cc:1398] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.042s user 0.004s sys 0.001s
I20240501 13:59:28.633307 25616 ts_tablet_manager.cc:1398] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
I20240501 13:59:28.633389 25602 raft_consensus.cc:686] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:28.633713 25606 raft_consensus.cc:686] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:28.633721 25616 tablet_bootstrap.cc:492] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:28.633796 25602 consensus_queue.cc:237] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:28.634096 25606 consensus_queue.cc:237] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.635739 25616 tablet_bootstrap.cc:654] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.636591 24546 catalog_manager.cc:5484] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
I20240501 13:59:28.636512 24548 catalog_manager.cc:5484] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
I20240501 13:59:28.638586 25616 tablet_bootstrap.cc:492] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:28.638785 25616 ts_tablet_manager.cc:1367] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 13:59:28.639672 25616 raft_consensus.cc:348] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.639958 25616 raft_consensus.cc:374] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.640045 25616 raft_consensus.cc:729] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.640343 25616 consensus_queue.cc:260] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.640566 25616 raft_consensus.cc:388] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:28.640673 25616 raft_consensus.cc:482] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:28.640771 25616 raft_consensus.cc:3037] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.643777 25616 raft_consensus.cc:504] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.643986 25616 leader_election.cc:302] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:28.644184 25616 leader_election.cc:288] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:28.644254 25620 raft_consensus.cc:2781] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.644385 25620 raft_consensus.cc:686] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:28.644791 25616 ts_tablet_manager.cc:1398] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240501 13:59:28.644717 25620 consensus_queue.cc:237] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:28.646795 24546 catalog_manager.cc:5484] T 9f89ef680f644d51baf2fd632a429af7 P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
W20240501 13:59:28.980311 25627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.980652 25627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.988158 25627 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 134.299 ms
time per row: 0.0671495 ms
I20240501 13:59:29.264127 24546 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:29.272768 24799 tablet_service.cc:1467] Processing CreateTablet for tablet c406608014684fc9adbdd5566d27b13a (DEFAULT_TABLE table=rebalance_test_table_2 [id=ba9000a5ac3743aab6f2067ed17d7d7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.272822 24669 tablet_service.cc:1467] Processing CreateTablet for tablet 6548d5ccc1454e53824d414b09e7e86e (DEFAULT_TABLE table=rebalance_test_table_2 [id=ba9000a5ac3743aab6f2067ed17d7d7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.272948 24668 tablet_service.cc:1467] Processing CreateTablet for tablet c757d90176b14f81b9804e2eab04c658 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ba9000a5ac3743aab6f2067ed17d7d7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.273269 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c406608014684fc9adbdd5566d27b13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.273280 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6548d5ccc1454e53824d414b09e7e86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.273519 24668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c757d90176b14f81b9804e2eab04c658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.279297 25669 tablet_bootstrap.cc:492] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:29.280426 25670 tablet_bootstrap.cc:492] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:29.280949 25669 tablet_bootstrap.cc:654] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.281863 25670 tablet_bootstrap.cc:654] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.282939 25669 tablet_bootstrap.cc:492] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:29.283092 25669 ts_tablet_manager.cc:1367] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:29.283725 25670 tablet_bootstrap.cc:492] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:29.283730 25669 raft_consensus.cc:348] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.283841 25670 ts_tablet_manager.cc:1367] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:29.283903 25669 raft_consensus.cc:374] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.284000 25669 raft_consensus.cc:729] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.284236 25669 consensus_queue.cc:260] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.284399 25669 raft_consensus.cc:388] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.284475 25669 raft_consensus.cc:482] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.284581 25669 raft_consensus.cc:3037] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.284504 25670 raft_consensus.cc:348] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.284649 25670 raft_consensus.cc:374] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.284739 25670 raft_consensus.cc:729] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.284941 25670 consensus_queue.cc:260] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.285146 25670 raft_consensus.cc:388] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.285230 25670 raft_consensus.cc:482] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.285331 25670 raft_consensus.cc:3037] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.287228 25670 raft_consensus.cc:504] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.287232 25669 raft_consensus.cc:504] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.287436 25670 leader_election.cc:302] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:29.287447 25669 leader_election.cc:302] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:29.287621 25669 leader_election.cc:288] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.287675 25620 raft_consensus.cc:2781] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.287671 25602 raft_consensus.cc:2781] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.288304 25669 ts_tablet_manager.cc:1398] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:29.288548 25669 tablet_bootstrap.cc:492] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:29.288942 25602 raft_consensus.cc:686] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:29.289252 25602 consensus_queue.cc:237] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.289942 25620 raft_consensus.cc:686] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:29.290066 25669 tablet_bootstrap.cc:654] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.290218 25620 consensus_queue.cc:237] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.292232 24546 catalog_manager.cc:5484] T c757d90176b14f81b9804e2eab04c658 P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
I20240501 13:59:29.293867 25670 leader_election.cc:288] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.294509 25670 ts_tablet_manager.cc:1398] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240501 13:59:29.294739 25669 tablet_bootstrap.cc:492] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:29.294888 25669 ts_tablet_manager.cc:1367] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 13:59:29.296957 24546 catalog_manager.cc:5484] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
I20240501 13:59:29.297158 25669 raft_consensus.cc:348] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.297338 25669 raft_consensus.cc:374] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.297403 25669 raft_consensus.cc:729] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.297641 25669 consensus_queue.cc:260] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.297811 25669 raft_consensus.cc:388] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.297904 25669 raft_consensus.cc:482] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.297999 25669 raft_consensus.cc:3037] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.300410 25669 raft_consensus.cc:504] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.300601 25669 leader_election.cc:302] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:29.300789 25669 leader_election.cc:288] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.300915 25602 raft_consensus.cc:2781] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.301106 25602 raft_consensus.cc:686] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:29.301277 25669 ts_tablet_manager.cc:1398] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240501 13:59:29.301411 25602 consensus_queue.cc:237] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.303670 24548 catalog_manager.cc:5484] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
W20240501 13:59:29.647804 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:29.648126 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:29.655771 25679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 116.543 ms
time per row: 0.0582715 ms
I20240501 13:59:29.912726 24548 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:29.923197 24669 tablet_service.cc:1467] Processing CreateTablet for tablet f7c2617355e1470fbd3a29c59830a8e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.923367 24799 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5f4f1d9e31406ca7bbebf02decc909 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.923657 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c2617355e1470fbd3a29c59830a8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.923794 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5f4f1d9e31406ca7bbebf02decc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.927807 24800 tablet_service.cc:1467] Processing CreateTablet for tablet 50555d4414914565859e6c59897edb0c (DEFAULT_TABLE table=rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:29.928177 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50555d4414914565859e6c59897edb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.940035 25719 tablet_bootstrap.cc:492] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:29.940929 25720 tablet_bootstrap.cc:492] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:29.941942 25719 tablet_bootstrap.cc:654] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.942492 25720 tablet_bootstrap.cc:654] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.943885 25719 tablet_bootstrap.cc:492] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:29.944018 25719 ts_tablet_manager.cc:1367] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:29.944299 25720 tablet_bootstrap.cc:492] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:29.944425 25720 ts_tablet_manager.cc:1367] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:29.944670 25719 raft_consensus.cc:348] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.944813 25719 raft_consensus.cc:374] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.944893 25719 raft_consensus.cc:729] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.945066 25720 raft_consensus.cc:348] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.945128 25719 consensus_queue.cc:260] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.945217 25720 raft_consensus.cc:374] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.945297 25719 raft_consensus.cc:388] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.945310 25720 raft_consensus.cc:729] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.945364 25719 raft_consensus.cc:482] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.945470 25719 raft_consensus.cc:3037] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.945530 25720 consensus_queue.cc:260] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.945703 25720 raft_consensus.cc:388] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.945781 25720 raft_consensus.cc:482] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.945883 25720 raft_consensus.cc:3037] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.947573 25719 raft_consensus.cc:504] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.947654 25720 raft_consensus.cc:504] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.947805 25719 leader_election.cc:302] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:29.947902 25720 leader_election.cc:302] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:29.948007 25719 leader_election.cc:288] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.948067 25602 raft_consensus.cc:2781] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.948117 25720 leader_election.cc:288] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.948289 25620 raft_consensus.cc:2781] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.949175 25720 ts_tablet_manager.cc:1398] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:29.949405 25720 tablet_bootstrap.cc:492] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:29.950759 25720 tablet_bootstrap.cc:654] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.952433 25720 tablet_bootstrap.cc:492] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:29.952584 25720 ts_tablet_manager.cc:1367] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:29.953215 25720 raft_consensus.cc:348] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.953392 25720 raft_consensus.cc:374] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.953490 25720 raft_consensus.cc:729] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.953759 25720 consensus_queue.cc:260] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.954013 25720 raft_consensus.cc:388] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.954139 25720 raft_consensus.cc:482] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.954259 25720 raft_consensus.cc:3037] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.956270 25720 raft_consensus.cc:504] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.956473 25720 leader_election.cc:302] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:29.956722 25720 leader_election.cc:288] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.956800 25724 raft_consensus.cc:2781] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.957295 25720 ts_tablet_manager.cc:1398] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 13:59:29.960480 25724 raft_consensus.cc:686] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:29.961194 25719 ts_tablet_manager.cc:1398] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20240501 13:59:29.961412 25620 raft_consensus.cc:686] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:29.961728 25620 consensus_queue.cc:237] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.960913 25724 consensus_queue.cc:237] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:29.961798 25602 raft_consensus.cc:686] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:29.962175 25602 consensus_queue.cc:237] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:29.965458 24548 catalog_manager.cc:5484] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
I20240501 13:59:29.965878 24543 catalog_manager.cc:5484] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
I20240501 13:59:29.970506 24548 catalog_manager.cc:5484] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
W20240501 13:59:30.309839 25733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:30.310169 25733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:30.317713 25733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 120.901 ms
time per row: 0.0604504 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 13:59:31.500125 24542 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54260:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:31.509009 24669 tablet_service.cc:1467] Processing CreateTablet for tablet 21494c260afc4690a2269129f4c4582f (DEFAULT_TABLE table=rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:31.509099 24800 tablet_service.cc:1467] Processing CreateTablet for tablet c82ba04565984d99967e1841826acce2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:31.509500 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21494c260afc4690a2269129f4c4582f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:31.509420 24668 tablet_service.cc:1467] Processing CreateTablet for tablet 840930a628ee422792b0c35c0278497a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:31.509507 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82ba04565984d99967e1841826acce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:31.509733 24668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840930a628ee422792b0c35c0278497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:31.515631 25775 tablet_bootstrap.cc:492] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Bootstrap starting.
I20240501 13:59:31.516479 25776 tablet_bootstrap.cc:492] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:31.517228 25775 tablet_bootstrap.cc:654] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:31.517709 25776 tablet_bootstrap.cc:654] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:31.519143 25775 tablet_bootstrap.cc:492] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: No bootstrap required, opened a new log
I20240501 13:59:31.519282 25775 ts_tablet_manager.cc:1367] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:31.519894 25775 raft_consensus.cc:348] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:31.520038 25775 raft_consensus.cc:374] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:31.520133 25775 raft_consensus.cc:729] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Initialized, Role: FOLLOWER
I20240501 13:59:31.520356 25775 consensus_queue.cc:260] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:31.520517 25775 raft_consensus.cc:388] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:31.520596 25775 raft_consensus.cc:482] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:31.520702 25775 raft_consensus.cc:3037] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:31.522542 25775 raft_consensus.cc:504] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:31.522799 25775 leader_election.cc:302] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:31.523244 25775 leader_election.cc:288] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:31.523384 25779 raft_consensus.cc:2781] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:31.524297 25775 ts_tablet_manager.cc:1398] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:31.524716 25779 raft_consensus.cc:686] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Leader. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:31.525002 25779 consensus_queue.cc:237] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:31.525509 25776 tablet_bootstrap.cc:492] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:31.525676 25776 ts_tablet_manager.cc:1367] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 13:59:31.526465 25776 raft_consensus.cc:348] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.526635 25776 raft_consensus.cc:374] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:31.526719 25776 raft_consensus.cc:729] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:31.526965 25776 consensus_queue.cc:260] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.527153 25776 raft_consensus.cc:388] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:31.527227 25776 raft_consensus.cc:482] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:31.527364 25776 raft_consensus.cc:3037] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:31.527467 24542 catalog_manager.cc:5484] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } }
I20240501 13:59:31.530398 25776 raft_consensus.cc:504] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.530619 25776 leader_election.cc:302] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:31.531008 25776 leader_election.cc:288] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:31.531168 25784 raft_consensus.cc:2781] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:31.531838 25784 raft_consensus.cc:686] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:31.531996 25776 ts_tablet_manager.cc:1398] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240501 13:59:31.532128 25784 consensus_queue.cc:237] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.532289 25776 tablet_bootstrap.cc:492] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Bootstrap starting.
I20240501 13:59:31.534353 25776 tablet_bootstrap.cc:654] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:31.534581 24542 catalog_manager.cc:5484] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
I20240501 13:59:31.536094 25776 tablet_bootstrap.cc:492] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: No bootstrap required, opened a new log
I20240501 13:59:31.536248 25776 ts_tablet_manager.cc:1367] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 13:59:31.536816 25776 raft_consensus.cc:348] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.536967 25776 raft_consensus.cc:374] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:31.537070 25776 raft_consensus.cc:729] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Initialized, Role: FOLLOWER
I20240501 13:59:31.537310 25776 consensus_queue.cc:260] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.537453 25776 raft_consensus.cc:388] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:31.537526 25776 raft_consensus.cc:482] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:31.537606 25776 raft_consensus.cc:3037] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:31.539587 25776 raft_consensus.cc:504] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.539764 25776 leader_election.cc:302] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d; no voters:
I20240501 13:59:31.539918 25776 leader_election.cc:288] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:31.539983 25784 raft_consensus.cc:2781] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:31.540167 25784 raft_consensus.cc:686] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Leader. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:31.540354 25776 ts_tablet_manager.cc:1398] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240501 13:59:31.540382 25784 consensus_queue.cc:237] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:31.542229 24543 catalog_manager.cc:5484] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65). New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } }
W20240501 13:59:31.876200 25791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:31.876549 25791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:31.884033 25791 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 119.159 ms
time per row: 0.0595794 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 13:59:33.071260 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:37985
--local_ip_for_outbound_sockets=127.22.33.67
--tserver_master_addrs=127.22.33.126:41135
--webserver_port=36717
--webserver_interface=127.22.33.67
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 13:59:33.412421 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:33.412735 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:33.412856 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:33.420536 25831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:33.420682 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:33.420850 25831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 13:59:33.430531 25831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:37985
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=36717
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:33.431982 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:33.433768 25831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:33.446585 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.447221 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.447970 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:33.448767 25831 server_base.cc:1034] running on GCE node
I20240501 13:59:33.449486 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:33.450518 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:33.451752 25831 hybrid_clock.cc:648] HybridClock initialized: now 1714571973451689 us; error 55 us; skew 500 ppm
I20240501 13:59:33.452059 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:33.457654 25831 webserver.cc:460] Webserver started at http://127.22.33.67:36717/ using document root <none> and password file <none>
I20240501 13:59:33.458443 25831 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:33.458566 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:33.464476 25831 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:59:33.467993 25845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.469611 25831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:33.469770 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b481b70e6d1f41408327ab1de26d7406"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:33.470672 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:33.485311 25831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:33.486161 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:33.486460 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:33.487427 25831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:33.489326 25852 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 13:59:33.494695 25831 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 13:59:33.494812 25831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 13:59:33.494956 25831 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 13:59:33.497994 25831 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 13:59:33.498100 25831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:33.498454 25852 tablet_bootstrap.cc:492] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap starting.
I20240501 13:59:33.518903 25852 log.cc:824] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:33.524051 25852 tablet_bootstrap.cc:492] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:33.524701 25852 tablet_bootstrap.cc:492] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap complete.
I20240501 13:59:33.526096 25852 ts_tablet_manager.cc:1367] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240501 13:59:33.535708 25852 raft_consensus.cc:348] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:33.537055 25852 raft_consensus.cc:729] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Initialized, Role: FOLLOWER
I20240501 13:59:33.537956 25852 consensus_queue.cc:260] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:33.538177 25852 raft_consensus.cc:388] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:33.538331 25852 raft_consensus.cc:482] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:33.538512 25852 raft_consensus.cc:3037] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:33.542608 25852 raft_consensus.cc:504] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:33.543268 25852 leader_election.cc:302] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b481b70e6d1f41408327ab1de26d7406; no voters:
I20240501 13:59:33.543969 25831 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:37985
I20240501 13:59:33.544036 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:37985 every 8 connection(s)
I20240501 13:59:33.544121 25852 leader_election.cc:288] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 13:59:33.544225 25960 raft_consensus.cc:2781] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:33.545764 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:33.546435 25852 ts_tablet_manager.cc:1398] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20240501 13:59:33.549316 25960 raft_consensus.cc:686] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [term 2 LEADER]: Becoming Leader. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Running, Role: LEADER
I20240501 13:59:33.549829 25960 consensus_queue.cc:237] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:33.550944 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 25831
I20240501 13:59:33.551683 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:44081
--local_ip_for_outbound_sockets=127.22.33.68
--tserver_master_addrs=127.22.33.126:41135
--webserver_port=38397
--webserver_interface=127.22.33.68
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:33.563409 25963 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:33.563807 25963 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:33.564877 25963 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:33.568080 24545 ts_manager.cc:194] Re-registered known tserver with Master: b481b70e6d1f41408327ab1de26d7406 (127.22.33.67:37985)
I20240501 13:59:33.568512 24545 catalog_manager.cc:5484] T 5e87402f3d5d42e5a5df4cccc16a20b2 P b481b70e6d1f41408327ab1de26d7406 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b481b70e6d1f41408327ab1de26d7406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:33.573702 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:40819
I20240501 13:59:33.576864 25963 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:33.889209 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:33.889544 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:33.889645 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:33.897367 25969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:33.897508 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:33.897622 25969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 13:59:33.907128 25969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:44081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=38397
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:33.908871 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:33.911266 25969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:33.925616 25978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.926111 25979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.929450 25981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:33.931404 25969 server_base.cc:1034] running on GCE node
I20240501 13:59:33.932185 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:33.933446 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:33.934674 25969 hybrid_clock.cc:648] HybridClock initialized: now 1714571973934614 us; error 66 us; skew 500 ppm
I20240501 13:59:33.935005 25969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:33.937458 25969 webserver.cc:460] Webserver started at http://127.22.33.68:38397/ using document root <none> and password file <none>
I20240501 13:59:33.938234 25969 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:33.938355 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:33.943791 25969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:59:33.946743 25987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.948068 25969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:33.948225 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "51db4831ba4348c3a502b54ae65eed48"
format_stamp: "Formatted at 2024-05-01 13:59:21 on dist-test-slave-3610"
I20240501 13:59:33.949152 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:33.996147 25969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:33.997092 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:33.997385 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:33.998340 25969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:33.999879 25969 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:33.999953 25969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.000072 25969 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:34.000119 25969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.032663 25969 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:44081
I20240501 13:59:34.032704 26099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:44081 every 8 connection(s)
I20240501 13:59:34.034143 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:34.042815 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 25969
I20240501 13:59:34.043516 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:34503
--local_ip_for_outbound_sockets=127.22.33.69
--tserver_master_addrs=127.22.33.126:41135
--webserver_port=45703
--webserver_interface=127.22.33.69
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:34.047037 26100 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:34.047353 26100 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:34.048243 26100 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:34.049852 24545 ts_manager.cc:194] Re-registered known tserver with Master: 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68:44081)
I20240501 13:59:34.050724 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:45755
W20240501 13:59:34.387149 26103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:34.387459 26103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:34.387547 26103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:34.394973 26103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:34.395104 26103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:34.395267 26103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 13:59:34.404417 26103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:34503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=45703
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:34.405800 26103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:34.407562 26103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:34.419732 26108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.420410 26111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.420902 26109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:34.421861 26103 server_base.cc:1034] running on GCE node
I20240501 13:59:34.422572 26103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:34.423808 26103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:34.425031 26103 hybrid_clock.cc:648] HybridClock initialized: now 1714571974424957 us; error 65 us; skew 500 ppm
I20240501 13:59:34.425447 26103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:34.428408 26103 webserver.cc:460] Webserver started at http://127.22.33.69:45703/ using document root <none> and password file <none>
I20240501 13:59:34.429280 26103 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:34.429405 26103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:34.434726 26103 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:34.437501 26117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.438795 26103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:34.438956 26103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ae9b30f780af481cbacd0566ab93e248"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:34.439762 26103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:34.467264 26103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:34.468140 26103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:34.468415 26103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:34.469347 26103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:34.470818 26103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:34.470896 26103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.471019 26103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:34.471072 26103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.501344 26103 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:34503
I20240501 13:59:34.501385 26229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:34503 every 8 connection(s)
I20240501 13:59:34.502859 26103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:34.507077 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 26103
I20240501 13:59:34.507710 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:37695
--local_ip_for_outbound_sockets=127.22.33.70
--tserver_master_addrs=127.22.33.126:41135
--webserver_port=35527
--webserver_interface=127.22.33.70
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:34.515566 26230 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:34.515872 26230 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:34.516583 26230 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:34.519076 24545 ts_manager.cc:194] Re-registered known tserver with Master: ae9b30f780af481cbacd0566ab93e248 (127.22.33.69:34503)
I20240501 13:59:34.521061 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:58739
W20240501 13:59:34.835844 26233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:34.836138 26233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:34.836230 26233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:34.843683 26233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:34.843832 26233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:34.843963 26233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 13:59:34.853386 26233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:37695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=35527
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:34.854815 26233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:34.856559 26233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:34.869094 26238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.869879 26241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.870426 26239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:34.871294 26233 server_base.cc:1034] running on GCE node
I20240501 13:59:34.871915 26233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:34.872824 26233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:34.874037 26233 hybrid_clock.cc:648] HybridClock initialized: now 1714571974873981 us; error 69 us; skew 500 ppm
I20240501 13:59:34.874387 26233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:34.876753 26233 webserver.cc:460] Webserver started at http://127.22.33.70:35527/ using document root <none> and password file <none>
I20240501 13:59:34.877521 26233 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:34.877645 26233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:34.883013 26233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:59:34.885936 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.887411 26233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:34.887598 26233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "35f855d8b43040f7843747f312c33239"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-3610"
I20240501 13:59:34.888469 26233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:34.908092 26233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:34.909011 26233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:34.909307 26233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:34.910287 26233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:34.912364 26233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:34.912446 26233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.912585 26233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:34.912640 26233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.945817 26233 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:37695
I20240501 13:59:34.945858 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:37695 every 8 connection(s)
I20240501 13:59:34.947328 26233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:34.948256 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 26233
I20240501 13:59:34.948964 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:37917
--local_ip_for_outbound_sockets=127.22.33.71
--tserver_master_addrs=127.22.33.126:41135
--webserver_port=39935
--webserver_interface=127.22.33.71
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 13:59:34.964206 26360 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:34.964551 26360 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:34.965345 26360 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:34.966787 24545 ts_manager.cc:194] Re-registered known tserver with Master: 35f855d8b43040f7843747f312c33239 (127.22.33.70:37695)
I20240501 13:59:34.967586 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:49459
I20240501 13:59:35.053232 26100 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
W20240501 13:59:35.280580 26363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:35.280910 26363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:35.281075 26363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:35.288597 26363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:35.288769 26363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:35.288939 26363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 13:59:35.298151 26363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:37917
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=39935
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:35.299574 26363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:35.301405 26363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:35.314541 26368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.315110 26371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.315920 26369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:35.316787 26363 server_base.cc:1034] running on GCE node
I20240501 13:59:35.317474 26363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:35.318564 26363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:35.319784 26363 hybrid_clock.cc:648] HybridClock initialized: now 1714571975319742 us; error 41 us; skew 500 ppm
I20240501 13:59:35.320118 26363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:35.322500 26363 webserver.cc:460] Webserver started at http://127.22.33.71:39935/ using document root <none> and password file <none>
I20240501 13:59:35.323233 26363 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:35.323343 26363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:35.328585 26363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 13:59:35.331429 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.332823 26363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:35.333011 26363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2c3d9ee3986341868e534aef0a98e8b0"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-3610"
I20240501 13:59:35.333859 26363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:35.354074 26363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:35.354950 26363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:35.355232 26363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:35.356163 26363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:35.357745 26363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:35.357815 26363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.357942 26363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:35.357987 26363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.389436 26363 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:37917
I20240501 13:59:35.389492 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:37917 every 8 connection(s)
I20240501 13:59:35.390883 26363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:35.399220 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 26363
I20240501 13:59:35.403606 26490 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:35.403910 26490 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:35.404886 26490 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:35.406499 24545 ts_manager.cc:194] Re-registered known tserver with Master: 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71:37917)
I20240501 13:59:35.407354 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:36729
I20240501 13:59:35.523545 26230 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:35.879611 24669 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:35.908198 26035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.911231 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.945413 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:35.946383 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:35.946702 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.964016 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.970487 26360 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:36.161334 24669 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:36.174424 26035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.178134 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:36.178987 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.206315 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.208344 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.208320 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:36.307910 24820 consensus_queue.cc:237] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
W20240501 13:59:36.317005 24754 consensus_peers.cc:485] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 -> Peer 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71:37917): Couldn't send request to peer 2c3d9ee3986341868e534aef0a98e8b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50555d4414914565859e6c59897edb0c. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:36.410457 26490 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:36.761608 26592 ts_tablet_manager.cc:902] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Initiating tablet copy from peer 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:36.763298 26592 tablet_copy_client.cc:308] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:40637
I20240501 13:59:36.770639 24840 tablet_copy_service.cc:140] P 85c2edb136794ec7a6e4352c57090d79: Received BeginTabletCopySession request for tablet 50555d4414914565859e6c59897edb0c from peer 2c3d9ee3986341868e534aef0a98e8b0 ({username='slave'} at 127.22.33.71:51447)
I20240501 13:59:36.770869 24840 tablet_copy_service.cc:161] P 85c2edb136794ec7a6e4352c57090d79: Beginning new tablet copy session on tablet 50555d4414914565859e6c59897edb0c from peer 2c3d9ee3986341868e534aef0a98e8b0 at {username='slave'} at 127.22.33.71:51447: session id = 2c3d9ee3986341868e534aef0a98e8b0-50555d4414914565859e6c59897edb0c
I20240501 13:59:36.773514 24840 tablet_copy_source_session.cc:205] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:36.778244 26592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50555d4414914565859e6c59897edb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.788264 26592 tablet_copy_client.cc:783] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:36.788565 26592 tablet_copy_client.cc:647] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:36.791155 26592 tablet_copy_client.cc:523] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:36.794337 26592 tablet_bootstrap.cc:492] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap starting.
I20240501 13:59:36.803414 26592 log.cc:824] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:36.835119 26592 tablet_bootstrap.cc:492] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:36.835687 26592 tablet_bootstrap.cc:492] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap complete.
I20240501 13:59:36.836107 26592 ts_tablet_manager.cc:1367] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Time spent bootstrapping tablet: real 0.042s user 0.031s sys 0.008s
I20240501 13:59:36.840757 26592 raft_consensus.cc:348] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:36.841606 26592 raft_consensus.cc:729] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3d9ee3986341868e534aef0a98e8b0, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.842051 26592 consensus_queue.cc:260] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:36.843616 26592 ts_tablet_manager.cc:1398] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240501 13:59:36.844714 24840 tablet_copy_service.cc:342] P 85c2edb136794ec7a6e4352c57090d79: Request end of tablet copy session 2c3d9ee3986341868e534aef0a98e8b0-50555d4414914565859e6c59897edb0c received from {username='slave'} at 127.22.33.71:51447
I20240501 13:59:36.844883 24840 tablet_copy_service.cc:433] P 85c2edb136794ec7a6e4352c57090d79: ending tablet copy session 2c3d9ee3986341868e534aef0a98e8b0-50555d4414914565859e6c59897edb0c on tablet 50555d4414914565859e6c59897edb0c with peer 2c3d9ee3986341868e534aef0a98e8b0
I20240501 13:59:37.288226 26445 raft_consensus.cc:1203] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:37.290060 26601 raft_consensus.cc:2932] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:37.290889 26445 raft_consensus.cc:2932] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:37.296332 24542 catalog_manager.cc:5484] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:37.328462 24689 consensus_queue.cc:237] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
W20240501 13:59:37.338167 24624 consensus_peers.cc:485] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d -> Peer ae9b30f780af481cbacd0566ab93e248 (127.22.33.69:34503): Couldn't send request to peer ae9b30f780af481cbacd0566ab93e248. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 840930a628ee422792b0c35c0278497a. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.795922 26628 ts_tablet_manager.cc:902] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:37.797766 26628 tablet_copy_client.cc:308] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:37.804086 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet 840930a628ee422792b0c35c0278497a from peer ae9b30f780af481cbacd0566ab93e248 ({username='slave'} at 127.22.33.69:55797)
I20240501 13:59:37.804412 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet 840930a628ee422792b0c35c0278497a from peer ae9b30f780af481cbacd0566ab93e248 at {username='slave'} at 127.22.33.69:55797: session id = ae9b30f780af481cbacd0566ab93e248-840930a628ee422792b0c35c0278497a
I20240501 13:59:37.807324 24709 tablet_copy_source_session.cc:205] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:37.812067 26628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840930a628ee422792b0c35c0278497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:37.822536 26628 tablet_copy_client.cc:783] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:37.822829 26628 tablet_copy_client.cc:647] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:37.825301 26628 tablet_copy_client.cc:523] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:37.828409 26628 tablet_bootstrap.cc:492] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Bootstrap starting.
I20240501 13:59:37.837488 26628 log.cc:824] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:37.868520 26628 tablet_bootstrap.cc:492] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:37.869138 26628 tablet_bootstrap.cc:492] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Bootstrap complete.
I20240501 13:59:37.869630 26628 ts_tablet_manager.cc:1367] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Time spent bootstrapping tablet: real 0.041s user 0.031s sys 0.012s
I20240501 13:59:37.873932 26628 raft_consensus.cc:348] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:37.874747 26628 raft_consensus.cc:729] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9b30f780af481cbacd0566ab93e248, State: Initialized, Role: FOLLOWER
I20240501 13:59:37.875173 26628 consensus_queue.cc:260] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:37.876546 26628 ts_tablet_manager.cc:1398] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 13:59:37.877588 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session ae9b30f780af481cbacd0566ab93e248-840930a628ee422792b0c35c0278497a received from {username='slave'} at 127.22.33.69:55797
I20240501 13:59:37.877760 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session ae9b30f780af481cbacd0566ab93e248-840930a628ee422792b0c35c0278497a on tablet 840930a628ee422792b0c35c0278497a with peer ae9b30f780af481cbacd0566ab93e248
I20240501 13:59:38.333860 26185 raft_consensus.cc:1203] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:38.335610 26638 raft_consensus.cc:2932] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:38.336539 26185 raft_consensus.cc:2932] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:38.339936 26638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:37944 (request call id 0) took 1012 ms. Trace:
I20240501 13:59:38.340165 26638 rpcz_store.cc:276] 0501 13:59:37.327401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:37.327472 (+ 71us) service_pool.cc:225] Handling call
0501 13:59:37.329723 (+ 2251us) log.cc:842] Serialized 257 byte log entry
0501 13:59:38.339878 (+1010155us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":74,"raft.queue_time_us":353,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":616,"spinlock_wait_cycles":91904,"thread_start_us":471,"threads_started":2,"wal-append.queue_time_us":328,"wal-append.run_cpu_time_us":679,"wal-append.run_wall_time_us":1000734}
I20240501 13:59:38.342459 24546 catalog_manager.cc:5484] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 reported cstate change: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:38.371606 24820 consensus_queue.cc:237] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
W20240501 13:59:38.379406 24754 consensus_peers.cc:485] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 -> Peer 35f855d8b43040f7843747f312c33239 (127.22.33.70:37695): Couldn't send request to peer 35f855d8b43040f7843747f312c33239. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 333670a3a08c4ef4b84def2560eda9c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:38.903375 26665 ts_tablet_manager.cc:902] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Initiating tablet copy from peer 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:38.905138 26665 tablet_copy_client.cc:308] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:40637
I20240501 13:59:38.911402 24840 tablet_copy_service.cc:140] P 85c2edb136794ec7a6e4352c57090d79: Received BeginTabletCopySession request for tablet 333670a3a08c4ef4b84def2560eda9c0 from peer 35f855d8b43040f7843747f312c33239 ({username='slave'} at 127.22.33.70:50353)
I20240501 13:59:38.911590 24840 tablet_copy_service.cc:161] P 85c2edb136794ec7a6e4352c57090d79: Beginning new tablet copy session on tablet 333670a3a08c4ef4b84def2560eda9c0 from peer 35f855d8b43040f7843747f312c33239 at {username='slave'} at 127.22.33.70:50353: session id = 35f855d8b43040f7843747f312c33239-333670a3a08c4ef4b84def2560eda9c0
I20240501 13:59:38.914201 24840 tablet_copy_source_session.cc:205] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:38.919281 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333670a3a08c4ef4b84def2560eda9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.930215 26665 tablet_copy_client.cc:783] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:38.930583 26665 tablet_copy_client.cc:647] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:38.932634 26665 tablet_copy_client.cc:523] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:38.935753 26665 tablet_bootstrap.cc:492] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Bootstrap starting.
I20240501 13:59:38.945565 26665 log.cc:824] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:38.980878 26665 tablet_bootstrap.cc:492] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:38.981472 26665 tablet_bootstrap.cc:492] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Bootstrap complete.
I20240501 13:59:38.981902 26665 ts_tablet_manager.cc:1367] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Time spent bootstrapping tablet: real 0.046s user 0.040s sys 0.007s
I20240501 13:59:38.986366 26665 raft_consensus.cc:348] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:38.987243 26665 raft_consensus.cc:729] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f855d8b43040f7843747f312c33239, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.987726 26665 consensus_queue.cc:260] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:38.989090 26665 ts_tablet_manager.cc:1398] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 13:59:38.990059 24840 tablet_copy_service.cc:342] P 85c2edb136794ec7a6e4352c57090d79: Request end of tablet copy session 35f855d8b43040f7843747f312c33239-333670a3a08c4ef4b84def2560eda9c0 received from {username='slave'} at 127.22.33.70:50353
I20240501 13:59:38.990197 24840 tablet_copy_service.cc:433] P 85c2edb136794ec7a6e4352c57090d79: ending tablet copy session 35f855d8b43040f7843747f312c33239-333670a3a08c4ef4b84def2560eda9c0 on tablet 333670a3a08c4ef4b84def2560eda9c0 with peer 35f855d8b43040f7843747f312c33239
I20240501 13:59:39.507738 26315 raft_consensus.cc:1203] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:39.509785 26677 raft_consensus.cc:2932] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:39.510424 26315 raft_consensus.cc:2932] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:39.512563 26677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35294 (request call id 0) took 1141 ms. Trace:
I20240501 13:59:39.512771 26677 rpcz_store.cc:276] 0501 13:59:38.371077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:38.371146 (+ 69us) service_pool.cc:225] Handling call
0501 13:59:38.372227 (+ 1081us) log.cc:842] Serialized 257 byte log entry
0501 13:59:39.512508 (+1140281us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":60,"raft.queue_time_us":38,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":197,"spinlock_wait_cycles":30464,"thread_start_us":179,"threads_started":1,"wal-append.queue_time_us":273,"wal-append.run_cpu_time_us":508,"wal-append.run_wall_time_us":1000516}
I20240501 13:59:39.516541 24542 catalog_manager.cc:5484] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 reported cstate change: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:39.547331 24689 consensus_queue.cc:237] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
W20240501 13:59:39.555035 24624 consensus_peers.cc:485] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d -> Peer 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68:44081): Couldn't send request to peer 51db4831ba4348c3a502b54ae65eed48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6548d5ccc1454e53824d414b09e7e86e. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:40.087657 26704 ts_tablet_manager.cc:902] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:40.089537 26704 tablet_copy_client.cc:308] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:40.096069 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet 6548d5ccc1454e53824d414b09e7e86e from peer 51db4831ba4348c3a502b54ae65eed48 ({username='slave'} at 127.22.33.68:45783)
I20240501 13:59:40.096249 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet 6548d5ccc1454e53824d414b09e7e86e from peer 51db4831ba4348c3a502b54ae65eed48 at {username='slave'} at 127.22.33.68:45783: session id = 51db4831ba4348c3a502b54ae65eed48-6548d5ccc1454e53824d414b09e7e86e
I20240501 13:59:40.098286 24709 tablet_copy_source_session.cc:205] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:40.102766 26704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6548d5ccc1454e53824d414b09e7e86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.113109 26704 tablet_copy_client.cc:783] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:40.113435 26704 tablet_copy_client.cc:647] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:40.115736 26704 tablet_copy_client.cc:523] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:40.119091 26704 tablet_bootstrap.cc:492] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Bootstrap starting.
I20240501 13:59:40.128523 26704 log.cc:824] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:40.161011 26704 tablet_bootstrap.cc:492] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:40.161593 26704 tablet_bootstrap.cc:492] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Bootstrap complete.
I20240501 13:59:40.162026 26704 ts_tablet_manager.cc:1367] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.004s
I20240501 13:59:40.166633 26704 raft_consensus.cc:348] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:40.167533 26704 raft_consensus.cc:729] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51db4831ba4348c3a502b54ae65eed48, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.167980 26704 consensus_queue.cc:260] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:40.169420 26704 ts_tablet_manager.cc:1398] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240501 13:59:40.170397 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session 51db4831ba4348c3a502b54ae65eed48-6548d5ccc1454e53824d414b09e7e86e received from {username='slave'} at 127.22.33.68:45783
I20240501 13:59:40.170507 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session 51db4831ba4348c3a502b54ae65eed48-6548d5ccc1454e53824d414b09e7e86e on tablet 6548d5ccc1454e53824d414b09e7e86e with peer 51db4831ba4348c3a502b54ae65eed48
I20240501 13:59:40.492416 26055 raft_consensus.cc:1203] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:40.494288 26713 raft_consensus.cc:2932] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:40.494946 26055 raft_consensus.cc:2932] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:40.499214 24542 catalog_manager.cc:5484] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:40.534396 24820 consensus_queue.cc:237] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
W20240501 13:59:40.542501 24753 consensus_peers.cc:485] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 -> Peer b481b70e6d1f41408327ab1de26d7406 (127.22.33.67:37985): Couldn't send request to peer b481b70e6d1f41408327ab1de26d7406. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c82ba04565984d99967e1841826acce2. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:40.986593 26738 ts_tablet_manager.cc:902] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Initiating tablet copy from peer 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:40.988207 26738 tablet_copy_client.cc:308] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:40637
I20240501 13:59:40.994566 24840 tablet_copy_service.cc:140] P 85c2edb136794ec7a6e4352c57090d79: Received BeginTabletCopySession request for tablet c82ba04565984d99967e1841826acce2 from peer b481b70e6d1f41408327ab1de26d7406 ({username='slave'} at 127.22.33.67:37751)
I20240501 13:59:40.994750 24840 tablet_copy_service.cc:161] P 85c2edb136794ec7a6e4352c57090d79: Beginning new tablet copy session on tablet c82ba04565984d99967e1841826acce2 from peer b481b70e6d1f41408327ab1de26d7406 at {username='slave'} at 127.22.33.67:37751: session id = b481b70e6d1f41408327ab1de26d7406-c82ba04565984d99967e1841826acce2
I20240501 13:59:40.996683 24840 tablet_copy_source_session.cc:205] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:40.998986 26738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82ba04565984d99967e1841826acce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.004623 26738 tablet_copy_client.cc:783] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:41.004894 26738 tablet_copy_client.cc:647] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:41.007025 26738 tablet_copy_client.cc:523] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:41.009804 26738 tablet_bootstrap.cc:492] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap starting.
I20240501 13:59:41.044870 26738 tablet_bootstrap.cc:492] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:41.045358 26738 tablet_bootstrap.cc:492] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Bootstrap complete.
I20240501 13:59:41.045598 26738 ts_tablet_manager.cc:1367] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.010s
I20240501 13:59:41.046351 26738 raft_consensus.cc:348] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:41.046778 26738 raft_consensus.cc:729] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.046934 26738 consensus_queue.cc:260] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:41.047667 26738 ts_tablet_manager.cc:1398] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 13:59:41.048672 24840 tablet_copy_service.cc:342] P 85c2edb136794ec7a6e4352c57090d79: Request end of tablet copy session b481b70e6d1f41408327ab1de26d7406-c82ba04565984d99967e1841826acce2 received from {username='slave'} at 127.22.33.67:37751
I20240501 13:59:41.048841 24840 tablet_copy_service.cc:433] P 85c2edb136794ec7a6e4352c57090d79: ending tablet copy session b481b70e6d1f41408327ab1de26d7406-c82ba04565984d99967e1841826acce2 on tablet c82ba04565984d99967e1841826acce2 with peer b481b70e6d1f41408327ab1de26d7406
I20240501 13:59:41.602069 25915 raft_consensus.cc:1203] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:41.603849 26747 raft_consensus.cc:2932] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b481b70e6d1f41408327ab1de26d7406 (127.22.33.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:41.604609 25915 raft_consensus.cc:2932] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b481b70e6d1f41408327ab1de26d7406 (127.22.33.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:41.606751 26747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35314 (request call id 0) took 1072 ms. Trace:
I20240501 13:59:41.606953 26747 rpcz_store.cc:276] 0501 13:59:40.533825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:40.533896 (+ 71us) service_pool.cc:225] Handling call
0501 13:59:40.535074 (+ 1178us) log.cc:842] Serialized 257 byte log entry
0501 13:59:41.606701 (+1071627us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":75,"raft.queue_time_us":50,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":159,"thread_start_us":189,"threads_started":1,"wal-append.queue_time_us":270,"wal-append.run_cpu_time_us":502,"wal-append.run_wall_time_us":1000567}
I20240501 13:59:41.609972 24542 catalog_manager.cc:5484] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: config changed from index -1 to 4, VOTER b481b70e6d1f41408327ab1de26d7406 (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:41.639272 24820 consensus_queue.cc:237] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
W20240501 13:59:41.641366 24754 consensus_peers.cc:485] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 -> Peer 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71:37917): Couldn't send request to peer 2c3d9ee3986341868e534aef0a98e8b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c406608014684fc9adbdd5566d27b13a. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:42.227275 26772 ts_tablet_manager.cc:902] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Initiating tablet copy from peer 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:42.228396 26772 tablet_copy_client.cc:308] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:40637
I20240501 13:59:42.229164 24840 tablet_copy_service.cc:140] P 85c2edb136794ec7a6e4352c57090d79: Received BeginTabletCopySession request for tablet c406608014684fc9adbdd5566d27b13a from peer 2c3d9ee3986341868e534aef0a98e8b0 ({username='slave'} at 127.22.33.71:51447)
I20240501 13:59:42.229329 24840 tablet_copy_service.cc:161] P 85c2edb136794ec7a6e4352c57090d79: Beginning new tablet copy session on tablet c406608014684fc9adbdd5566d27b13a from peer 2c3d9ee3986341868e534aef0a98e8b0 at {username='slave'} at 127.22.33.71:51447: session id = 2c3d9ee3986341868e534aef0a98e8b0-c406608014684fc9adbdd5566d27b13a
I20240501 13:59:42.231513 24840 tablet_copy_source_session.cc:205] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:42.232761 26772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c406608014684fc9adbdd5566d27b13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.237139 26772 tablet_copy_client.cc:783] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:42.237368 26772 tablet_copy_client.cc:647] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:42.239180 26772 tablet_copy_client.cc:523] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:42.241724 26772 tablet_bootstrap.cc:492] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap starting.
I20240501 13:59:42.277063 26772 tablet_bootstrap.cc:492] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:42.277653 26772 tablet_bootstrap.cc:492] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Bootstrap complete.
I20240501 13:59:42.277951 26772 ts_tablet_manager.cc:1367] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Time spent bootstrapping tablet: real 0.036s user 0.036s sys 0.000s
I20240501 13:59:42.278873 26772 raft_consensus.cc:348] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:42.279247 26772 raft_consensus.cc:729] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3d9ee3986341868e534aef0a98e8b0, State: Initialized, Role: FOLLOWER
I20240501 13:59:42.279417 26772 consensus_queue.cc:260] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:42.280213 26772 ts_tablet_manager.cc:1398] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:42.281194 24840 tablet_copy_service.cc:342] P 85c2edb136794ec7a6e4352c57090d79: Request end of tablet copy session 2c3d9ee3986341868e534aef0a98e8b0-c406608014684fc9adbdd5566d27b13a received from {username='slave'} at 127.22.33.71:51447
I20240501 13:59:42.281350 24840 tablet_copy_service.cc:433] P 85c2edb136794ec7a6e4352c57090d79: ending tablet copy session 2c3d9ee3986341868e534aef0a98e8b0-c406608014684fc9adbdd5566d27b13a on tablet c406608014684fc9adbdd5566d27b13a with peer 2c3d9ee3986341868e534aef0a98e8b0
I20240501 13:59:42.757496 26445 raft_consensus.cc:1203] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:42.758926 26780 raft_consensus.cc:2932] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:42.759603 26445 raft_consensus.cc:2932] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:42.762022 26780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35334 (request call id 0) took 1123 ms. Trace:
I20240501 13:59:42.762212 26780 rpcz_store.cc:276] 0501 13:59:41.638712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:41.638789 (+ 77us) service_pool.cc:225] Handling call
0501 13:59:41.639818 (+ 1029us) log.cc:842] Serialized 257 byte log entry
0501 13:59:42.761962 (+1122144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":184,"thread_start_us":254,"threads_started":1,"wal-append.queue_time_us":362,"wal-append.run_cpu_time_us":601,"wal-append.run_wall_time_us":1000671}
I20240501 13:59:42.764431 24542 catalog_manager.cc:5484] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 reported cstate change: config changed from index -1 to 4, VOTER 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:42.799328 24689 consensus_queue.cc:237] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
W20240501 13:59:42.801658 24624 consensus_peers.cc:485] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d -> Peer ae9b30f780af481cbacd0566ab93e248 (127.22.33.69:34503): Couldn't send request to peer ae9b30f780af481cbacd0566ab93e248. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb1d595a67cb40e195e9db6841978c0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.279626 26805 ts_tablet_manager.cc:902] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:43.280797 26805 tablet_copy_client.cc:308] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:43.281584 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet bb1d595a67cb40e195e9db6841978c0d from peer ae9b30f780af481cbacd0566ab93e248 ({username='slave'} at 127.22.33.69:55797)
I20240501 13:59:43.281744 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet bb1d595a67cb40e195e9db6841978c0d from peer ae9b30f780af481cbacd0566ab93e248 at {username='slave'} at 127.22.33.69:55797: session id = ae9b30f780af481cbacd0566ab93e248-bb1d595a67cb40e195e9db6841978c0d
I20240501 13:59:43.283756 24709 tablet_copy_source_session.cc:205] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:43.285048 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1d595a67cb40e195e9db6841978c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.289589 26805 tablet_copy_client.cc:783] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:43.289862 26805 tablet_copy_client.cc:647] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:43.291707 26805 tablet_copy_client.cc:523] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:43.294481 26805 tablet_bootstrap.cc:492] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Bootstrap starting.
I20240501 13:59:43.324535 26805 tablet_bootstrap.cc:492] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:43.324919 26805 tablet_bootstrap.cc:492] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Bootstrap complete.
I20240501 13:59:43.325173 26805 ts_tablet_manager.cc:1367] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.005s
I20240501 13:59:43.325955 26805 raft_consensus.cc:348] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:43.326331 26805 raft_consensus.cc:729] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9b30f780af481cbacd0566ab93e248, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.326488 26805 consensus_queue.cc:260] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:43.327153 26805 ts_tablet_manager.cc:1398] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:43.328009 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session ae9b30f780af481cbacd0566ab93e248-bb1d595a67cb40e195e9db6841978c0d received from {username='slave'} at 127.22.33.69:55797
I20240501 13:59:43.328168 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session ae9b30f780af481cbacd0566ab93e248-bb1d595a67cb40e195e9db6841978c0d on tablet bb1d595a67cb40e195e9db6841978c0d with peer ae9b30f780af481cbacd0566ab93e248
I20240501 13:59:43.887363 26185 raft_consensus.cc:1203] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:43.889154 26813 raft_consensus.cc:2932] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:43.889703 26185 raft_consensus.cc:2932] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:43.892472 26813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50476 (request call id 0) took 1093 ms. Trace:
I20240501 13:59:43.892692 26813 rpcz_store.cc:276] 0501 13:59:42.798654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:42.798741 (+ 87us) service_pool.cc:225] Handling call
0501 13:59:42.799963 (+ 1222us) log.cc:842] Serialized 257 byte log entry
0501 13:59:43.892350 (+1092387us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":58,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":225,"spinlock_wait_cycles":69504,"thread_start_us":237,"threads_started":1,"wal-append.queue_time_us":356,"wal-append.run_cpu_time_us":543,"wal-append.run_wall_time_us":1000610}
I20240501 13:59:43.895540 24546 catalog_manager.cc:5484] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: config changed from index -1 to 4, VOTER ae9b30f780af481cbacd0566ab93e248 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:43.926199 24689 consensus_queue.cc:237] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
W20240501 13:59:43.934718 24623 consensus_peers.cc:485] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d -> Peer 35f855d8b43040f7843747f312c33239 (127.22.33.70:37695): Couldn't send request to peer 35f855d8b43040f7843747f312c33239. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7c2617355e1470fbd3a29c59830a8e6. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:44.447160 26840 ts_tablet_manager.cc:902] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:44.448526 26840 tablet_copy_client.cc:308] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:44.454895 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet f7c2617355e1470fbd3a29c59830a8e6 from peer 35f855d8b43040f7843747f312c33239 ({username='slave'} at 127.22.33.70:44247)
I20240501 13:59:44.455086 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet f7c2617355e1470fbd3a29c59830a8e6 from peer 35f855d8b43040f7843747f312c33239 at {username='slave'} at 127.22.33.70:44247: session id = 35f855d8b43040f7843747f312c33239-f7c2617355e1470fbd3a29c59830a8e6
I20240501 13:59:44.457149 24709 tablet_copy_source_session.cc:205] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.458459 26840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c2617355e1470fbd3a29c59830a8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.463086 26840 tablet_copy_client.cc:783] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.463330 26840 tablet_copy_client.cc:647] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.465377 26840 tablet_copy_client.cc:523] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.470269 26840 tablet_bootstrap.cc:492] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Bootstrap starting.
I20240501 13:59:44.505249 26840 tablet_bootstrap.cc:492] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:44.505684 26840 tablet_bootstrap.cc:492] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Bootstrap complete.
I20240501 13:59:44.505973 26840 ts_tablet_manager.cc:1367] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Time spent bootstrapping tablet: real 0.036s user 0.031s sys 0.005s
I20240501 13:59:44.506779 26840 raft_consensus.cc:348] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:44.507190 26840 raft_consensus.cc:729] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f855d8b43040f7843747f312c33239, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.507381 26840 consensus_queue.cc:260] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:44.509620 26840 ts_tablet_manager.cc:1398] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 13:59:44.510581 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session 35f855d8b43040f7843747f312c33239-f7c2617355e1470fbd3a29c59830a8e6 received from {username='slave'} at 127.22.33.70:44247
I20240501 13:59:44.510752 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session 35f855d8b43040f7843747f312c33239-f7c2617355e1470fbd3a29c59830a8e6 on tablet f7c2617355e1470fbd3a29c59830a8e6 with peer 35f855d8b43040f7843747f312c33239
I20240501 13:59:45.012166 26315 raft_consensus.cc:1203] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:45.013870 26850 raft_consensus.cc:2932] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:45.014611 26315 raft_consensus.cc:2932] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:45.017726 26850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50490 (request call id 0) took 1092 ms. Trace:
I20240501 13:59:45.017946 26850 rpcz_store.cc:276] 0501 13:59:43.925641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:43.925723 (+ 82us) service_pool.cc:225] Handling call
0501 13:59:43.926931 (+ 1208us) log.cc:842] Serialized 257 byte log entry
0501 13:59:45.017660 (+1090729us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":89,"raft.queue_time_us":58,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"spinlock_wait_cycles":31488,"thread_start_us":262,"threads_started":1,"wal-append.queue_time_us":385,"wal-append.run_cpu_time_us":582,"wal-append.run_wall_time_us":1000638}
I20240501 13:59:45.020412 24546 catalog_manager.cc:5484] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 reported cstate change: config changed from index -1 to 4, VOTER 35f855d8b43040f7843747f312c33239 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:45.056946 24689 consensus_queue.cc:237] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
W20240501 13:59:45.059301 24624 consensus_peers.cc:485] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d -> Peer 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68:44081): Couldn't send request to peer 51db4831ba4348c3a502b54ae65eed48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b35dc4ce0bf144d19068653fdfd1e1b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:45.667696 26875 ts_tablet_manager.cc:902] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:45.669026 26875 tablet_copy_client.cc:308] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:45.669725 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet b35dc4ce0bf144d19068653fdfd1e1b8 from peer 51db4831ba4348c3a502b54ae65eed48 ({username='slave'} at 127.22.33.68:45783)
I20240501 13:59:45.669898 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet b35dc4ce0bf144d19068653fdfd1e1b8 from peer 51db4831ba4348c3a502b54ae65eed48 at {username='slave'} at 127.22.33.68:45783: session id = 51db4831ba4348c3a502b54ae65eed48-b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 13:59:45.671850 24709 tablet_copy_source_session.cc:205] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:45.673115 26875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35dc4ce0bf144d19068653fdfd1e1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.677901 26875 tablet_copy_client.cc:783] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:45.678217 26875 tablet_copy_client.cc:647] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:45.680006 26875 tablet_copy_client.cc:523] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:45.682734 26875 tablet_bootstrap.cc:492] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Bootstrap starting.
I20240501 13:59:45.713949 26875 tablet_bootstrap.cc:492] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:45.714365 26875 tablet_bootstrap.cc:492] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Bootstrap complete.
I20240501 13:59:45.714666 26875 ts_tablet_manager.cc:1367] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Time spent bootstrapping tablet: real 0.032s user 0.032s sys 0.000s
I20240501 13:59:45.715603 26875 raft_consensus.cc:348] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:45.715952 26875 raft_consensus.cc:729] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51db4831ba4348c3a502b54ae65eed48, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.716122 26875 consensus_queue.cc:260] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:45.717064 26875 ts_tablet_manager.cc:1398] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:45.718161 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session 51db4831ba4348c3a502b54ae65eed48-b35dc4ce0bf144d19068653fdfd1e1b8 received from {username='slave'} at 127.22.33.68:45783
I20240501 13:59:45.718312 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session 51db4831ba4348c3a502b54ae65eed48-b35dc4ce0bf144d19068653fdfd1e1b8 on tablet b35dc4ce0bf144d19068653fdfd1e1b8 with peer 51db4831ba4348c3a502b54ae65eed48
I20240501 13:59:46.078330 26055 raft_consensus.cc:1203] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:46.079267 26850 raft_consensus.cc:2932] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:46.079957 26055 raft_consensus.cc:2932] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:46.083101 26850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50516 (request call id 0) took 1026 ms. Trace:
I20240501 13:59:46.083329 26850 rpcz_store.cc:276] 0501 13:59:45.056335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:45.056404 (+ 69us) service_pool.cc:225] Handling call
0501 13:59:45.057694 (+ 1290us) log.cc:842] Serialized 257 byte log entry
0501 13:59:46.083021 (+1025327us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":82,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":200,"thread_start_us":258,"threads_started":1,"wal-append.queue_time_us":403,"wal-append.run_cpu_time_us":499,"wal-append.run_wall_time_us":1000555}
I20240501 13:59:46.086199 24543 catalog_manager.cc:5484] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: config changed from index -1 to 4, VOTER 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:46.193732 24669 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:46.220467 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.230749 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.251717 26035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.272715 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.287245 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:46.292438 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.355451 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b35dc4ce0bf144d19068653fdfd1e1b8"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50552
I20240501 13:59:46.355748 24689 raft_consensus.cc:595] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:46.451011 24669 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:46.451650 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.461925 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.485289 26035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.498282 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.510756 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.542470 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:46.618840 26979 raft_consensus.cc:979] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower 51db4831ba4348c3a502b54ae65eed48 to start an election
I20240501 13:59:46.619014 26979 raft_consensus.cc:1067] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer 51db4831ba4348c3a502b54ae65eed48 to start an election
I20240501 13:59:46.619918 26055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b35dc4ce0bf144d19068653fdfd1e1b8"
dest_uuid: "51db4831ba4348c3a502b54ae65eed48"
from {username='slave'} at 127.22.33.65:51729
I20240501 13:59:46.620196 26055 raft_consensus.cc:482] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:46.620388 26055 raft_consensus.cc:3037] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:46.622506 26055 raft_consensus.cc:504] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:46.623842 26055 leader_election.cc:288] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:46.624512 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35dc4ce0bf144d19068653fdfd1e1b8" candidate_uuid: "51db4831ba4348c3a502b54ae65eed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:46.624785 24689 raft_consensus.cc:3032] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:46.624878 24689 raft_consensus.cc:729] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:46.625226 24689 consensus_queue.cc:260] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:46.625598 24689 raft_consensus.cc:3037] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:46.627902 24689 raft_consensus.cc:2454] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51db4831ba4348c3a502b54ae65eed48 in term 2.
I20240501 13:59:46.628521 25988 leader_election.cc:302] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, 51db4831ba4348c3a502b54ae65eed48; no voters:
I20240501 13:59:46.629233 26980 raft_consensus.cc:2781] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:46.629681 26980 raft_consensus.cc:686] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 2 LEADER]: Becoming Leader. State: Replica: 51db4831ba4348c3a502b54ae65eed48, State: Running, Role: LEADER
I20240501 13:59:46.630265 26980 consensus_queue.cc:237] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:46.634016 24543 catalog_manager.cc:5484] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68). New cstate: current_term: 2 leader_uuid: "51db4831ba4348c3a502b54ae65eed48" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:46.704622 24669 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.712081 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:46.717298 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.717785 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:46.719965 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.721752 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.745342 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.814602 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7c2617355e1470fbd3a29c59830a8e6"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50602
I20240501 13:59:46.814828 24689 raft_consensus.cc:595] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:46.908393 24669 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.921474 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.925048 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.925488 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:46.944339 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:46.949966 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:46.959424 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.019088 24689 raft_consensus.cc:1261] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer 51db4831ba4348c3a502b54ae65eed48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:47.020244 26980 consensus_queue.cc:1035] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:47.082311 26979 raft_consensus.cc:979] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower 35f855d8b43040f7843747f312c33239 to start an election
I20240501 13:59:47.082466 26979 raft_consensus.cc:1067] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer 35f855d8b43040f7843747f312c33239 to start an election
I20240501 13:59:47.083323 26315 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7c2617355e1470fbd3a29c59830a8e6"
dest_uuid: "35f855d8b43040f7843747f312c33239"
from {username='slave'} at 127.22.33.65:44761
I20240501 13:59:47.083611 26315 raft_consensus.cc:482] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:47.083799 26315 raft_consensus.cc:3037] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.086216 26315 raft_consensus.cc:504] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:47.087486 26315 leader_election.cc:288] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:47.088013 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c2617355e1470fbd3a29c59830a8e6" candidate_uuid: "35f855d8b43040f7843747f312c33239" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:47.088199 24689 raft_consensus.cc:3032] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:47.088290 24689 raft_consensus.cc:729] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:47.088559 24689 consensus_queue.cc:260] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:47.088817 24689 raft_consensus.cc:3037] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.090832 24689 raft_consensus.cc:2454] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f855d8b43040f7843747f312c33239 in term 2.
I20240501 13:59:47.091359 26248 leader_election.cc:302] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, 35f855d8b43040f7843747f312c33239; no voters:
I20240501 13:59:47.092027 27061 raft_consensus.cc:2781] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:47.092505 27061 raft_consensus.cc:686] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 2 LEADER]: Becoming Leader. State: Replica: 35f855d8b43040f7843747f312c33239, State: Running, Role: LEADER
I20240501 13:59:47.093086 27061 consensus_queue.cc:237] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:47.099401 24545 catalog_manager.cc:5484] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to 35f855d8b43040f7843747f312c33239 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "35f855d8b43040f7843747f312c33239" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:47.164873 24669 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:47.176860 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.177102 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.192026 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.201016 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.201857 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:47.207212 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.279152 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb1d595a67cb40e195e9db6841978c0d"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50626
I20240501 13:59:47.279363 24689 raft_consensus.cc:595] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:47.380137 24669 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:47.388821 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.389660 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.390583 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.406965 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:47.417416 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.419636 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.599707 24669 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:47.606808 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.616803 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.623402 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.643573 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:47.660815 26165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.672763 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.715334 24689 raft_consensus.cc:1261] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer 35f855d8b43040f7843747f312c33239: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:47.716292 27164 consensus_queue.cc:1035] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 13:59:47.749410 26850 raft_consensus.cc:979] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower ae9b30f780af481cbacd0566ab93e248 to start an election
I20240501 13:59:47.749596 26850 raft_consensus.cc:1067] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer ae9b30f780af481cbacd0566ab93e248 to start an election
I20240501 13:59:47.750514 26185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb1d595a67cb40e195e9db6841978c0d"
dest_uuid: "ae9b30f780af481cbacd0566ab93e248"
from {username='slave'} at 127.22.33.65:58703
I20240501 13:59:47.750842 26185 raft_consensus.cc:482] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:47.751045 26185 raft_consensus.cc:3037] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.753806 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6548d5ccc1454e53824d414b09e7e86e"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49618
I20240501 13:59:47.753844 26185 raft_consensus.cc:504] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:47.754014 24689 raft_consensus.cc:595] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:47.755388 26185 leader_election.cc:288] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:47.755937 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1d595a67cb40e195e9db6841978c0d" candidate_uuid: "ae9b30f780af481cbacd0566ab93e248" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:47.756214 24689 raft_consensus.cc:3032] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:47.756332 24689 raft_consensus.cc:729] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:47.756664 24689 consensus_queue.cc:260] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:47.757004 24689 raft_consensus.cc:3037] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.759706 24689 raft_consensus.cc:2454] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9b30f780af481cbacd0566ab93e248 in term 2.
I20240501 13:59:47.760339 26118 leader_election.cc:302] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, ae9b30f780af481cbacd0566ab93e248; no voters:
I20240501 13:59:47.761241 27181 raft_consensus.cc:2781] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:47.761736 27181 raft_consensus.cc:686] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 2 LEADER]: Becoming Leader. State: Replica: ae9b30f780af481cbacd0566ab93e248, State: Running, Role: LEADER
I20240501 13:59:47.762336 27181 consensus_queue.cc:237] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:47.766855 24544 catalog_manager.cc:5484] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to ae9b30f780af481cbacd0566ab93e248 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "ae9b30f780af481cbacd0566ab93e248" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:47.846578 24669 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:47.859467 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.877099 26165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.878170 26035 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:47.878856 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:47.879962 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:47.881983 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.063246 27170 raft_consensus.cc:979] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower 51db4831ba4348c3a502b54ae65eed48 to start an election
I20240501 13:59:48.063459 27170 raft_consensus.cc:1067] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer 51db4831ba4348c3a502b54ae65eed48 to start an election
I20240501 13:59:48.064467 26055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6548d5ccc1454e53824d414b09e7e86e"
dest_uuid: "51db4831ba4348c3a502b54ae65eed48"
from {username='slave'} at 127.22.33.65:51729
I20240501 13:59:48.064751 26055 raft_consensus.cc:482] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:48.064865 26055 raft_consensus.cc:3037] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.067521 26055 raft_consensus.cc:504] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:48.068332 26055 leader_election.cc:288] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:48.068686 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6548d5ccc1454e53824d414b09e7e86e" candidate_uuid: "51db4831ba4348c3a502b54ae65eed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:48.068948 24689 raft_consensus.cc:3032] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:48.069080 24689 raft_consensus.cc:729] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:48.069358 24689 consensus_queue.cc:260] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:48.069677 24689 raft_consensus.cc:3037] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.074904 24689 raft_consensus.cc:2454] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51db4831ba4348c3a502b54ae65eed48 in term 2.
I20240501 13:59:48.075449 25988 leader_election.cc:302] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, 51db4831ba4348c3a502b54ae65eed48; no voters:
I20240501 13:59:48.075867 27160 raft_consensus.cc:2781] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:48.076022 27160 raft_consensus.cc:686] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 2 LEADER]: Becoming Leader. State: Replica: 51db4831ba4348c3a502b54ae65eed48, State: Running, Role: LEADER
I20240501 13:59:48.076314 27160 consensus_queue.cc:237] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:48.078935 24543 catalog_manager.cc:5484] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to 51db4831ba4348c3a502b54ae65eed48 (127.22.33.68). New cstate: current_term: 2 leader_uuid: "51db4831ba4348c3a502b54ae65eed48" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:48.097687 24669 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:48.111948 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.114190 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:48.120147 24800 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:48.154227 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.170135 26165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.184500 25894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.228520 24820 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c82ba04565984d99967e1841826acce2"
dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38692
I20240501 13:59:48.228782 24820 raft_consensus.cc:595] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:48.242969 24689 raft_consensus.cc:1261] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer ae9b30f780af481cbacd0566ab93e248: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:48.243806 27181 consensus_queue.cc:1035] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:48.252856 27264 raft_consensus.cc:979] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: : Instructing follower b481b70e6d1f41408327ab1de26d7406 to start an election
I20240501 13:59:48.253067 27264 raft_consensus.cc:1067] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Signalling peer b481b70e6d1f41408327ab1de26d7406 to start an election
I20240501 13:59:48.254086 25915 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82ba04565984d99967e1841826acce2"
dest_uuid: "b481b70e6d1f41408327ab1de26d7406"
from {username='slave'} at 127.22.33.66:54247
I20240501 13:59:48.254365 25915 raft_consensus.cc:482] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:48.254493 25915 raft_consensus.cc:3037] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.257009 25915 raft_consensus.cc:504] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:48.258034 25915 leader_election.cc:288] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 2 election: Requested vote from peers 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:48.258814 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82ba04565984d99967e1841826acce2" candidate_uuid: "b481b70e6d1f41408327ab1de26d7406" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
I20240501 13:59:48.259109 24820 raft_consensus.cc:3032] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:48.259217 24820 raft_consensus.cc:729] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:48.259475 24820 consensus_queue.cc:260] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:48.259883 24820 raft_consensus.cc:3037] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.261968 24820 raft_consensus.cc:2454] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b481b70e6d1f41408327ab1de26d7406 in term 2.
I20240501 13:59:48.262596 25847 leader_election.cc:302] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 85c2edb136794ec7a6e4352c57090d79, b481b70e6d1f41408327ab1de26d7406; no voters:
I20240501 13:59:48.263278 27265 raft_consensus.cc:2781] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:48.263749 27265 raft_consensus.cc:686] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 2 LEADER]: Becoming Leader. State: Replica: b481b70e6d1f41408327ab1de26d7406, State: Running, Role: LEADER
I20240501 13:59:48.264103 27265 consensus_queue.cc:237] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:48.267351 24543 catalog_manager.cc:5484] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 reported cstate change: term changed from 1 to 2, leader changed from 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) to b481b70e6d1f41408327ab1de26d7406 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "b481b70e6d1f41408327ab1de26d7406" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:48.319465 24669 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:48.350049 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.350916 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:48.350930 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.366930 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:48.367062 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.374361 26165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.457280 24689 raft_consensus.cc:1261] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer 51db4831ba4348c3a502b54ae65eed48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:48.457888 27160 consensus_queue.cc:1035] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:48.569752 24669 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:48.569913 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:48.584357 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.586513 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.592139 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:48.611248 26165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.629977 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.684952 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "840930a628ee422792b0c35c0278497a"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49672
I20240501 13:59:48.685200 24689 raft_consensus.cc:595] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:48.773710 24669 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:48.800100 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.801270 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:48.803500 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.805491 26165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.810077 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:48.827191 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:48.853153 24820 raft_consensus.cc:1261] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Refusing update from remote peer b481b70e6d1f41408327ab1de26d7406: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:48.854274 27372 consensus_queue.cc:1035] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:48.882593 27377 raft_consensus.cc:979] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower ae9b30f780af481cbacd0566ab93e248 to start an election
I20240501 13:59:48.882870 27170 raft_consensus.cc:1067] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer ae9b30f780af481cbacd0566ab93e248 to start an election
I20240501 13:59:48.883667 26185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "840930a628ee422792b0c35c0278497a"
dest_uuid: "ae9b30f780af481cbacd0566ab93e248"
from {username='slave'} at 127.22.33.65:58703
I20240501 13:59:48.883925 26185 raft_consensus.cc:482] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:48.884042 26185 raft_consensus.cc:3037] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.886893 26185 raft_consensus.cc:504] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:48.888238 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840930a628ee422792b0c35c0278497a" candidate_uuid: "ae9b30f780af481cbacd0566ab93e248" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:48.888468 24689 raft_consensus.cc:3032] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:48.888540 24689 raft_consensus.cc:729] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:48.888829 24689 consensus_queue.cc:260] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:48.889065 24689 raft_consensus.cc:3037] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:48.891896 24689 raft_consensus.cc:2454] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9b30f780af481cbacd0566ab93e248 in term 2.
I20240501 13:59:48.892809 26118 leader_election.cc:302] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, ae9b30f780af481cbacd0566ab93e248; no voters:
I20240501 13:59:48.893182 27182 raft_consensus.cc:2781] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:48.894203 26185 leader_election.cc:288] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:48.897325 27182 raft_consensus.cc:686] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 2 LEADER]: Becoming Leader. State: Replica: ae9b30f780af481cbacd0566ab93e248, State: Running, Role: LEADER
I20240501 13:59:48.897751 27182 consensus_queue.cc:237] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:48.900219 24545 catalog_manager.cc:5484] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to ae9b30f780af481cbacd0566ab93e248 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "ae9b30f780af481cbacd0566ab93e248" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:49.045202 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:49.047629 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.061445 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.063755 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:49.073112 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.076560 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:49.083669 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.164521 24820 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "333670a3a08c4ef4b84def2560eda9c0"
dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38732
I20240501 13:59:49.164780 24820 raft_consensus.cc:595] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:49.255519 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.267133 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:49.270999 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:49.272804 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.298449 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:49.318650 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.326068 24689 raft_consensus.cc:1261] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer ae9b30f780af481cbacd0566ab93e248: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:49.327060 27182 consensus_queue.cc:1035] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:49.340808 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.483136 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.492830 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.505265 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:49.509435 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:49.531944 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.545037 24800 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:49.564285 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.585489 27508 raft_consensus.cc:979] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: : Instructing follower 35f855d8b43040f7843747f312c33239 to start an election
I20240501 13:59:49.585753 27264 raft_consensus.cc:1067] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Signalling peer 35f855d8b43040f7843747f312c33239 to start an election
I20240501 13:59:49.586592 26315 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "333670a3a08c4ef4b84def2560eda9c0"
dest_uuid: "35f855d8b43040f7843747f312c33239"
from {username='slave'} at 127.22.33.66:40347
I20240501 13:59:49.586853 26315 raft_consensus.cc:482] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:49.586956 26315 raft_consensus.cc:3037] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:49.590245 26315 raft_consensus.cc:504] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:49.591490 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333670a3a08c4ef4b84def2560eda9c0" candidate_uuid: "35f855d8b43040f7843747f312c33239" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
I20240501 13:59:49.591718 24820 raft_consensus.cc:3032] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:49.591812 24820 raft_consensus.cc:729] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:49.592092 24820 consensus_queue.cc:260] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:49.592311 24820 raft_consensus.cc:3037] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:49.595268 24820 raft_consensus.cc:2454] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f855d8b43040f7843747f312c33239 in term 2.
I20240501 13:59:49.595971 26249 leader_election.cc:302] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f855d8b43040f7843747f312c33239, 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:49.596311 27404 raft_consensus.cc:2781] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:49.597342 26315 leader_election.cc:288] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [CANDIDATE]: Term 2 election: Requested vote from peers 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:49.612442 27404 raft_consensus.cc:686] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 2 LEADER]: Becoming Leader. State: Replica: 35f855d8b43040f7843747f312c33239, State: Running, Role: LEADER
I20240501 13:59:49.612838 27404 consensus_queue.cc:237] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:49.619275 24545 catalog_manager.cc:5484] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 reported cstate change: term changed from 1 to 2, leader changed from 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) to 35f855d8b43040f7843747f312c33239 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "35f855d8b43040f7843747f312c33239" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:49.643234 24820 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50555d4414914565859e6c59897edb0c"
dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38762
I20240501 13:59:49.643440 24820 raft_consensus.cc:595] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:49.660323 27508 raft_consensus.cc:979] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: : Instructing follower 2c3d9ee3986341868e534aef0a98e8b0 to start an election
I20240501 13:59:49.660508 27508 raft_consensus.cc:1067] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Signalling peer 2c3d9ee3986341868e534aef0a98e8b0 to start an election
I20240501 13:59:49.661298 26445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50555d4414914565859e6c59897edb0c"
dest_uuid: "2c3d9ee3986341868e534aef0a98e8b0"
from {username='slave'} at 127.22.33.66:60615
I20240501 13:59:49.661604 26445 raft_consensus.cc:482] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:49.661783 26445 raft_consensus.cc:3037] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:49.664908 26445 raft_consensus.cc:504] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:49.666597 26445 leader_election.cc:288] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [CANDIDATE]: Term 2 election: Requested vote from peers 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:49.667196 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50555d4414914565859e6c59897edb0c" candidate_uuid: "2c3d9ee3986341868e534aef0a98e8b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
I20240501 13:59:49.667443 24820 raft_consensus.cc:3032] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:49.667585 24820 raft_consensus.cc:729] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:49.667896 24820 consensus_queue.cc:260] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:49.668200 24820 raft_consensus.cc:3037] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:49.670195 24820 raft_consensus.cc:2454] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3d9ee3986341868e534aef0a98e8b0 in term 2.
I20240501 13:59:49.670733 26379 leader_election.cc:302] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3d9ee3986341868e534aef0a98e8b0, 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:49.671422 27525 raft_consensus.cc:2781] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:49.671962 27525 raft_consensus.cc:686] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 LEADER]: Becoming Leader. State: Replica: 2c3d9ee3986341868e534aef0a98e8b0, State: Running, Role: LEADER
I20240501 13:59:49.672444 27525 consensus_queue.cc:237] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:49.676380 24545 catalog_manager.cc:5484] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 reported cstate change: term changed from 1 to 2, leader changed from 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) to 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71). New cstate: current_term: 2 leader_uuid: "2c3d9ee3986341868e534aef0a98e8b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:49.743714 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.760390 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.760533 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.765319 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.769956 24800 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:49.778661 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:49.794019 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.960774 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.989632 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.006443 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:50.010344 24800 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:50.010982 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.012735 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.027777 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.095867 24820 raft_consensus.cc:1261] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Refusing update from remote peer 2c3d9ee3986341868e534aef0a98e8b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:50.096702 27525 consensus_queue.cc:1035] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:50.109172 24820 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c406608014684fc9adbdd5566d27b13a"
dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38810
I20240501 13:59:50.109346 24820 raft_consensus.cc:595] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:50.159895 27264 raft_consensus.cc:979] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: : Instructing follower 2c3d9ee3986341868e534aef0a98e8b0 to start an election
I20240501 13:59:50.160085 27264 raft_consensus.cc:1067] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Signalling peer 2c3d9ee3986341868e534aef0a98e8b0 to start an election
I20240501 13:59:50.160758 26445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c406608014684fc9adbdd5566d27b13a"
dest_uuid: "2c3d9ee3986341868e534aef0a98e8b0"
from {username='slave'} at 127.22.33.66:60615
I20240501 13:59:50.161036 26445 raft_consensus.cc:482] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:50.161162 26445 raft_consensus.cc:3037] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:50.163365 26445 raft_consensus.cc:504] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:50.163872 26445 leader_election.cc:288] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [CANDIDATE]: Term 2 election: Requested vote from peers 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:50.164835 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c406608014684fc9adbdd5566d27b13a" candidate_uuid: "2c3d9ee3986341868e534aef0a98e8b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
I20240501 13:59:50.165050 24820 raft_consensus.cc:3032] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:50.165153 24820 raft_consensus.cc:729] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:50.165366 24820 consensus_queue.cc:260] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:50.165616 24820 raft_consensus.cc:3037] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:50.167456 24820 raft_consensus.cc:2454] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3d9ee3986341868e534aef0a98e8b0 in term 2.
I20240501 13:59:50.167847 26379 leader_election.cc:302] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c3d9ee3986341868e534aef0a98e8b0, 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:50.168259 27526 raft_consensus.cc:2781] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:50.168388 27526 raft_consensus.cc:686] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 LEADER]: Becoming Leader. State: Replica: 2c3d9ee3986341868e534aef0a98e8b0, State: Running, Role: LEADER
I20240501 13:59:50.168730 27526 consensus_queue.cc:237] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:50.171590 24541 catalog_manager.cc:5484] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 reported cstate change: term changed from 1 to 2, leader changed from 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) to 2c3d9ee3986341868e534aef0a98e8b0 (127.22.33.71). New cstate: current_term: 2 leader_uuid: "2c3d9ee3986341868e534aef0a98e8b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:50.176795 24820 raft_consensus.cc:1261] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Refusing update from remote peer 35f855d8b43040f7843747f312c33239: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:50.177414 27509 consensus_queue.cc:1035] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:50.217787 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.225850 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.229749 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.230679 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.236119 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.245390 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.270159 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.622629 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.636936 24820 raft_consensus.cc:1261] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Refusing update from remote peer 2c3d9ee3986341868e534aef0a98e8b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:50.637682 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.639150 27526 consensus_queue.cc:1035] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:50.640643 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.648079 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.658727 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.704259 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.705521 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.801501 26055 consensus_queue.cc:237] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:50.802794 27160 raft_consensus.cc:2932] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:50.807333 24545 catalog_manager.cc:5484] T b35dc4ce0bf144d19068653fdfd1e1b8 P 51db4831ba4348c3a502b54ae65eed48 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "51db4831ba4348c3a502b54ae65eed48" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:50.812813 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet b35dc4ce0bf144d19068653fdfd1e1b8 with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 13:59:50.813745 27700 tablet_replica.cc:330] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 13:59:50.814241 27700 raft_consensus.cc:2229] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:50.814514 27700 raft_consensus.cc:2258] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:50.815953 27700 ts_tablet_manager.cc:1875] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:50.822108 27700 ts_tablet_manager.cc:1888] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:50.822253 27700 log.cc:1196] T b35dc4ce0bf144d19068653fdfd1e1b8 P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 13:59:50.823165 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet b35dc4ce0bf144d19068653fdfd1e1b8 (table rebalance_test_table_1 [id=3aa4500548d34d888c0efe6d86f95330]) successfully deleted
I20240501 13:59:50.923517 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.938743 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.965996 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.967200 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.985978 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:50.986447 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.988579 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.101609 26315 consensus_queue.cc:237] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:51.102972 27623 raft_consensus.cc:2932] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:51.108175 24543 catalog_manager.cc:5484] T f7c2617355e1470fbd3a29c59830a8e6 P 35f855d8b43040f7843747f312c33239 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "35f855d8b43040f7843747f312c33239" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:51.112164 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c2617355e1470fbd3a29c59830a8e6 with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 13:59:51.112584 27700 tablet_replica.cc:330] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 13:59:51.112876 27700 raft_consensus.cc:2229] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:51.113171 27700 raft_consensus.cc:2258] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:51.114207 27700 ts_tablet_manager.cc:1875] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:51.121058 27700 ts_tablet_manager.cc:1888] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:51.121223 27700 log.cc:1196] T f7c2617355e1470fbd3a29c59830a8e6 P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7c2617355e1470fbd3a29c59830a8e6
I20240501 13:59:51.122094 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet f7c2617355e1470fbd3a29c59830a8e6 (table rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]) successfully deleted
I20240501 13:59:51.215786 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.242952 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.249142 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.254031 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.263939 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.275250 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.275336 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.367166 26185 consensus_queue.cc:237] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:51.368506 27182 raft_consensus.cc:2932] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:51.373656 24546 catalog_manager.cc:5484] T bb1d595a67cb40e195e9db6841978c0d P ae9b30f780af481cbacd0566ab93e248 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "ae9b30f780af481cbacd0566ab93e248" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:51.377964 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1d595a67cb40e195e9db6841978c0d with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 13:59:51.378357 27700 tablet_replica.cc:330] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 13:59:51.378545 27700 raft_consensus.cc:2229] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:51.378736 27700 raft_consensus.cc:2258] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:51.379354 27700 ts_tablet_manager.cc:1875] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:51.384666 27700 ts_tablet_manager.cc:1888] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:51.384824 27700 log.cc:1196] T bb1d595a67cb40e195e9db6841978c0d P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb1d595a67cb40e195e9db6841978c0d
I20240501 13:59:51.385653 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet bb1d595a67cb40e195e9db6841978c0d (table rebalance_test_table_0 [id=2435645932f448f5a82db7bdb4911b7f]) successfully deleted
I20240501 13:59:51.471011 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.482270 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.493649 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.526429 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.527798 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.550290 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.550590 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.637578 26055 consensus_queue.cc:237] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } }
I20240501 13:59:51.638877 27160 raft_consensus.cc:2932] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:51.643492 24543 catalog_manager.cc:5484] T 6548d5ccc1454e53824d414b09e7e86e P 51db4831ba4348c3a502b54ae65eed48 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "51db4831ba4348c3a502b54ae65eed48" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51db4831ba4348c3a502b54ae65eed48" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 44081 } } }
I20240501 13:59:51.647332 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet 6548d5ccc1454e53824d414b09e7e86e with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 13:59:51.647675 27700 tablet_replica.cc:330] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 13:59:51.647862 27700 raft_consensus.cc:2229] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:51.648049 27700 raft_consensus.cc:2258] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:51.648700 27700 ts_tablet_manager.cc:1875] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:51.653539 27700 ts_tablet_manager.cc:1888] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:51.653715 27700 log.cc:1196] T 6548d5ccc1454e53824d414b09e7e86e P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/6548d5ccc1454e53824d414b09e7e86e
I20240501 13:59:51.654474 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet 6548d5ccc1454e53824d414b09e7e86e (table rebalance_test_table_2 [id=ba9000a5ac3743aab6f2067ed17d7d7e]) successfully deleted
I20240501 13:59:51.756462 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.760711 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.774385 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.774587 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.792439 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.794195 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:51.794409 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:51.895445 25915 consensus_queue.cc:237] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } }
I20240501 13:59:51.896754 27671 raft_consensus.cc:2932] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:51.901981 24543 catalog_manager.cc:5484] T c82ba04565984d99967e1841826acce2 P b481b70e6d1f41408327ab1de26d7406 reported cstate change: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New cstate: current_term: 2 leader_uuid: "b481b70e6d1f41408327ab1de26d7406" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b481b70e6d1f41408327ab1de26d7406" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 37985 } } }
I20240501 13:59:51.905615 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet c82ba04565984d99967e1841826acce2 with delete_type TABLET_DATA_TOMBSTONED (TS 85c2edb136794ec7a6e4352c57090d79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39914
I20240501 13:59:51.906531 27915 tablet_replica.cc:330] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: stopping tablet replica
I20240501 13:59:51.907109 27915 raft_consensus.cc:2229] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:51.907431 27915 raft_consensus.cc:2258] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:51.909106 27915 ts_tablet_manager.cc:1875] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:51.915443 27915 ts_tablet_manager.cc:1888] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:51.915627 27915 log.cc:1196] T c82ba04565984d99967e1841826acce2 P 85c2edb136794ec7a6e4352c57090d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/c82ba04565984d99967e1841826acce2
I20240501 13:59:51.916724 24533 catalog_manager.cc:4830] TS 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637): tablet c82ba04565984d99967e1841826acce2 (table rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]) successfully deleted
I20240501 13:59:52.011742 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.027891 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.031323 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.033607 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.040743 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.057096 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.057106 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.158196 26185 consensus_queue.cc:237] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } }
I20240501 13:59:52.159509 27182 raft_consensus.cc:2932] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:52.164093 24545 catalog_manager.cc:5484] T 840930a628ee422792b0c35c0278497a P ae9b30f780af481cbacd0566ab93e248 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "ae9b30f780af481cbacd0566ab93e248" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae9b30f780af481cbacd0566ab93e248" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 34503 } } }
I20240501 13:59:52.167981 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet 840930a628ee422792b0c35c0278497a with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 13:59:52.168594 27972 tablet_replica.cc:330] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 13:59:52.168795 27972 raft_consensus.cc:2229] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.169040 27972 raft_consensus.cc:2258] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.169647 27972 ts_tablet_manager.cc:1875] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.174438 27972 ts_tablet_manager.cc:1888] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:52.174584 27972 log.cc:1196] T 840930a628ee422792b0c35c0278497a P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/840930a628ee422792b0c35c0278497a
I20240501 13:59:52.175520 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet 840930a628ee422792b0c35c0278497a (table rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]) successfully deleted
I20240501 13:59:52.272683 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.287295 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.291721 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.292434 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.293557 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.299444 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.307590 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.413758 26315 consensus_queue.cc:237] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } }
I20240501 13:59:52.415040 28007 raft_consensus.cc:2932] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:52.419574 24541 catalog_manager.cc:5484] T 333670a3a08c4ef4b84def2560eda9c0 P 35f855d8b43040f7843747f312c33239 reported cstate change: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New cstate: current_term: 2 leader_uuid: "35f855d8b43040f7843747f312c33239" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "35f855d8b43040f7843747f312c33239" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37695 } } }
I20240501 13:59:52.424167 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet 333670a3a08c4ef4b84def2560eda9c0 with delete_type TABLET_DATA_TOMBSTONED (TS 85c2edb136794ec7a6e4352c57090d79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39914
I20240501 13:59:52.425529 28027 tablet_replica.cc:330] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: stopping tablet replica
I20240501 13:59:52.425796 28027 raft_consensus.cc:2229] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.426028 28027 raft_consensus.cc:2258] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.426860 28027 ts_tablet_manager.cc:1875] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.433704 28027 ts_tablet_manager.cc:1888] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:52.433840 28027 log.cc:1196] T 333670a3a08c4ef4b84def2560eda9c0 P 85c2edb136794ec7a6e4352c57090d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/333670a3a08c4ef4b84def2560eda9c0
I20240501 13:59:52.434715 24533 catalog_manager.cc:4830] TS 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637): tablet 333670a3a08c4ef4b84def2560eda9c0 (table rebalance_test_table_1 [id=3aa4500548d34d888c0efe6d86f95330]) successfully deleted
I20240501 13:59:52.541270 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.556548 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.560966 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.565598 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.567495 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.567718 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.570293 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.680330 26445 consensus_queue.cc:237] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:52.681872 27884 raft_consensus.cc:2932] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:52.686758 24545 catalog_manager.cc:5484] T 50555d4414914565859e6c59897edb0c P 2c3d9ee3986341868e534aef0a98e8b0 reported cstate change: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New cstate: current_term: 2 leader_uuid: "2c3d9ee3986341868e534aef0a98e8b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:52.690964 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet 50555d4414914565859e6c59897edb0c with delete_type TABLET_DATA_TOMBSTONED (TS 85c2edb136794ec7a6e4352c57090d79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39914
I20240501 13:59:52.691397 28027 tablet_replica.cc:330] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: stopping tablet replica
I20240501 13:59:52.691610 28027 raft_consensus.cc:2229] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.691845 28027 raft_consensus.cc:2258] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.692592 28027 ts_tablet_manager.cc:1875] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.698278 28027 ts_tablet_manager.cc:1888] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:52.698427 28027 log.cc:1196] T 50555d4414914565859e6c59897edb0c P 85c2edb136794ec7a6e4352c57090d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/50555d4414914565859e6c59897edb0c
I20240501 13:59:52.699330 24533 catalog_manager.cc:4830] TS 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637): tablet 50555d4414914565859e6c59897edb0c (table rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]) successfully deleted
I20240501 13:59:52.793818 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.804411 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.806663 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.807319 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.807439 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.810302 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.836165 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:52.937985 26445 consensus_queue.cc:237] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } }
I20240501 13:59:52.939394 27884 raft_consensus.cc:2932] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:52.944447 24545 catalog_manager.cc:5484] T c406608014684fc9adbdd5566d27b13a P 2c3d9ee3986341868e534aef0a98e8b0 reported cstate change: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New cstate: current_term: 2 leader_uuid: "2c3d9ee3986341868e534aef0a98e8b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3d9ee3986341868e534aef0a98e8b0" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 37917 } } }
I20240501 13:59:52.948913 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet c406608014684fc9adbdd5566d27b13a with delete_type TABLET_DATA_TOMBSTONED (TS 85c2edb136794ec7a6e4352c57090d79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39914
I20240501 13:59:52.949292 28027 tablet_replica.cc:330] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: stopping tablet replica
I20240501 13:59:52.949473 28027 raft_consensus.cc:2229] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.949668 28027 raft_consensus.cc:2258] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.950335 28027 ts_tablet_manager.cc:1875] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.955420 28027 ts_tablet_manager.cc:1888] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 13:59:52.955561 28027 log.cc:1196] T c406608014684fc9adbdd5566d27b13a P 85c2edb136794ec7a6e4352c57090d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/c406608014684fc9adbdd5566d27b13a
I20240501 13:59:52.956329 24533 catalog_manager.cc:4830] TS 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637): tablet c406608014684fc9adbdd5566d27b13a (table rebalance_test_table_2 [id=ba9000a5ac3743aab6f2067ed17d7d7e]) successfully deleted
I20240501 13:59:53.322252 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.329421 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.331120 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.344556 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.345831 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.356206 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:53.381861 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:53.546708 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:53.546759 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.558938 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.567906 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.597254 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.606043 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:53.618050 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.380031 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.381614 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:54.409912 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.454317 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.457283 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:54.464385 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.477372 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.655786 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:54.672617 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.685475 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.689705 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:54.689750 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.694375 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.707635 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:54.991124 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:55.004489 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.006333 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:55.007926 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.012773 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.026271 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.042449 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ad3ae472198240bf908bfe32012c5a39 | 127.22.33.126:41135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.33.84:41329 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0e31c1e6d8f843558a29d08245b4c93d | 127.22.33.65:36237 | HEALTHY | <none> | 3 | 0
2c3d9ee3986341868e534aef0a98e8b0 | 127.22.33.71:37917 | HEALTHY | <none> | 2 | 0
35f855d8b43040f7843747f312c33239 | 127.22.33.70:37695 | HEALTHY | <none> | 2 | 0
51db4831ba4348c3a502b54ae65eed48 | 127.22.33.68:44081 | HEALTHY | <none> | 2 | 0
85c2edb136794ec7a6e4352c57090d79 | 127.22.33.66:40637 | HEALTHY | <none> | 3 | 0
ae9b30f780af481cbacd0566ab93e248 | 127.22.33.69:34503 | HEALTHY | <none> | 2 | 0
b481b70e6d1f41408327ab1de26d7406 | 127.22.33.67:37985 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.33.65 | experimental | 127.22.33.65:36237
local_ip_for_outbound_sockets | 127.22.33.66 | experimental | 127.22.33.66:40637
local_ip_for_outbound_sockets | 127.22.33.67 | experimental | 127.22.33.67:37985
local_ip_for_outbound_sockets | 127.22.33.68 | experimental | 127.22.33.68:44081
local_ip_for_outbound_sockets | 127.22.33.69 | experimental | 127.22.33.69:34503
local_ip_for_outbound_sockets | 127.22.33.70 | experimental | 127.22.33.70:37695
local_ip_for_outbound_sockets | 127.22.33.71 | experimental | 127.22.33.71:37917
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.33.65:36237
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.33.66:40637
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.33.67:37985
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.33.68:44081
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.33.69:34503
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.33.70:37695
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.33.71:37917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.33.84:41329 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 13:59:55.106797 22661 log_verifier.cc:126] Checking tablet 04a8ca79aa05482c898141739d0ebd7d
I20240501 13:59:55.121886 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04a8ca79aa05482c898141739d0ebd7d
I20240501 13:59:55.122045 22661 log_verifier.cc:126] Checking tablet 21494c260afc4690a2269129f4c4582f
I20240501 13:59:55.134227 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 21494c260afc4690a2269129f4c4582f
I20240501 13:59:55.134356 22661 log_verifier.cc:126] Checking tablet 333670a3a08c4ef4b84def2560eda9c0
I20240501 13:59:55.147203 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 333670a3a08c4ef4b84def2560eda9c0
I20240501 13:59:55.147341 22661 log_verifier.cc:126] Checking tablet 50555d4414914565859e6c59897edb0c
I20240501 13:59:55.159736 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 50555d4414914565859e6c59897edb0c
I20240501 13:59:55.159863 22661 log_verifier.cc:126] Checking tablet 5e87402f3d5d42e5a5df4cccc16a20b2
I20240501 13:59:55.167788 22661 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5e87402f3d5d42e5a5df4cccc16a20b2
I20240501 13:59:55.167915 22661 log_verifier.cc:126] Checking tablet 6548d5ccc1454e53824d414b09e7e86e
I20240501 13:59:55.180573 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6548d5ccc1454e53824d414b09e7e86e
I20240501 13:59:55.180694 22661 log_verifier.cc:126] Checking tablet 7b5f4f1d9e31406ca7bbebf02decc909
I20240501 13:59:55.192826 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7b5f4f1d9e31406ca7bbebf02decc909
I20240501 13:59:55.192947 22661 log_verifier.cc:126] Checking tablet 840930a628ee422792b0c35c0278497a
I20240501 13:59:55.205785 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 840930a628ee422792b0c35c0278497a
I20240501 13:59:55.205904 22661 log_verifier.cc:126] Checking tablet 9f89ef680f644d51baf2fd632a429af7
I20240501 13:59:55.217653 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9f89ef680f644d51baf2fd632a429af7
I20240501 13:59:55.217764 22661 log_verifier.cc:126] Checking tablet b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 13:59:55.230384 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 13:59:55.230504 22661 log_verifier.cc:126] Checking tablet bb1d595a67cb40e195e9db6841978c0d
I20240501 13:59:55.242933 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bb1d595a67cb40e195e9db6841978c0d
I20240501 13:59:55.243045 22661 log_verifier.cc:126] Checking tablet c406608014684fc9adbdd5566d27b13a
I20240501 13:59:55.255402 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c406608014684fc9adbdd5566d27b13a
I20240501 13:59:55.255517 22661 log_verifier.cc:126] Checking tablet c757d90176b14f81b9804e2eab04c658
I20240501 13:59:55.268486 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c757d90176b14f81b9804e2eab04c658
I20240501 13:59:55.268608 22661 log_verifier.cc:126] Checking tablet c82ba04565984d99967e1841826acce2
I20240501 13:59:55.281539 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c82ba04565984d99967e1841826acce2
I20240501 13:59:55.281666 22661 log_verifier.cc:126] Checking tablet c8caad758e6243e48008fe469425fcff
I20240501 13:59:55.294356 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c8caad758e6243e48008fe469425fcff
I20240501 13:59:55.294473 22661 log_verifier.cc:126] Checking tablet f7c2617355e1470fbd3a29c59830a8e6
I20240501 13:59:55.307436 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f7c2617355e1470fbd3a29c59830a8e6
I20240501 13:59:55.310689 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.72:0
--local_ip_for_outbound_sockets=127.22.33.72
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--builtin_ntp_servers=127.22.33.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 13:59:55.651525 28357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:55.651847 28357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:55.651986 28357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:55.660907 28357 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 13:59:55.661078 28357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:55.661268 28357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.72
I20240501 13:59:55.670686 28357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:41135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.72
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:55.671981 28357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:55.673748 28357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:55.687018 28363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.687860 28362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.688482 28365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:55.689245 28357 server_base.cc:1034] running on GCE node
I20240501 13:59:55.689851 28357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:55.690790 28357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:55.691994 28357 hybrid_clock.cc:648] HybridClock initialized: now 1714571995691940 us; error 48 us; skew 500 ppm
I20240501 13:59:55.692301 28357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:55.699015 28357 webserver.cc:460] Webserver started at http://127.22.33.72:44873/ using document root <none> and password file <none>
I20240501 13:59:55.699797 28357 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:55.699923 28357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:55.700249 28357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:55.702978 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "68eab98270d449dbbccecfc3a4cf8766"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-3610"
I20240501 13:59:55.703740 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "68eab98270d449dbbccecfc3a4cf8766"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-3610"
I20240501 13:59:55.709007 28357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:55.712492 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:55.713843 28357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:55.714016 28357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "68eab98270d449dbbccecfc3a4cf8766"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-3610"
I20240501 13:59:55.714216 28357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:55.742354 28357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:55.743261 28357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:55.743556 28357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:55.744536 28357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:55.746117 28357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:55.746218 28357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:55.746333 28357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:55.746402 28357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:55.782939 28357 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.72:40329
I20240501 13:59:55.782984 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.72:40329 every 8 connection(s)
I20240501 13:59:55.784323 28357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 13:59:55.790431 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 28357
I20240501 13:59:55.790652 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 13:59:55.801808 28484 heartbeater.cc:347] Connected to a master server at 127.22.33.126:41135
I20240501 13:59:55.802147 28484 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:55.802908 28484 heartbeater.cc:511] Master 127.22.33.126:41135 requested a full tablet report, sending...
I20240501 13:59:55.804303 24545 ts_manager.cc:194] Registered new tserver with Master: 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72:40329)
I20240501 13:59:55.805130 24545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.72:40351
I20240501 13:59:56.294840 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:56.296954 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.299407 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.321419 28419 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:56.353266 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.363471 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.377419 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:56.410542 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.591892 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:56.604260 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.619648 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.621445 28419 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:56.645264 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.648680 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.650171 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:56.660681 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.757400 24689 consensus_queue.cc:237] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
W20240501 13:59:56.766713 24625 consensus_peers.cc:485] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d -> Peer 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72:40329): Couldn't send request to peer 68eab98270d449dbbccecfc3a4cf8766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21494c260afc4690a2269129f4c4582f. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:56.807617 28484 heartbeater.cc:503] Master 127.22.33.126:41135 was elected leader, sending a full tablet report...
I20240501 13:59:57.328058 28590 ts_tablet_manager.cc:902] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Initiating tablet copy from peer 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:57.329831 28590 tablet_copy_client.cc:308] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:36237
I20240501 13:59:57.336400 24709 tablet_copy_service.cc:140] P 0e31c1e6d8f843558a29d08245b4c93d: Received BeginTabletCopySession request for tablet 21494c260afc4690a2269129f4c4582f from peer 68eab98270d449dbbccecfc3a4cf8766 ({username='slave'} at 127.22.33.72:49707)
I20240501 13:59:57.336618 24709 tablet_copy_service.cc:161] P 0e31c1e6d8f843558a29d08245b4c93d: Beginning new tablet copy session on tablet 21494c260afc4690a2269129f4c4582f from peer 68eab98270d449dbbccecfc3a4cf8766 at {username='slave'} at 127.22.33.72:49707: session id = 68eab98270d449dbbccecfc3a4cf8766-21494c260afc4690a2269129f4c4582f
I20240501 13:59:57.338701 24709 tablet_copy_source_session.cc:205] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:57.343092 28590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21494c260afc4690a2269129f4c4582f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:57.352692 28590 tablet_copy_client.cc:783] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:57.353032 28590 tablet_copy_client.cc:647] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:57.355324 28590 tablet_copy_client.cc:523] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:57.358598 28590 tablet_bootstrap.cc:492] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap starting.
I20240501 13:59:57.368111 28590 log.cc:824] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:57.404619 28590 tablet_bootstrap.cc:492] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:57.405304 28590 tablet_bootstrap.cc:492] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap complete.
I20240501 13:59:57.405815 28590 ts_tablet_manager.cc:1367] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Time spent bootstrapping tablet: real 0.047s user 0.038s sys 0.009s
I20240501 13:59:57.411222 28590 raft_consensus.cc:348] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } }
I20240501 13:59:57.412030 28590 raft_consensus.cc:729] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eab98270d449dbbccecfc3a4cf8766, State: Initialized, Role: FOLLOWER
I20240501 13:59:57.412443 28590 consensus_queue.cc:260] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:57.414041 28590 ts_tablet_manager.cc:1398] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20240501 13:59:57.415124 24709 tablet_copy_service.cc:342] P 0e31c1e6d8f843558a29d08245b4c93d: Request end of tablet copy session 68eab98270d449dbbccecfc3a4cf8766-21494c260afc4690a2269129f4c4582f received from {username='slave'} at 127.22.33.72:49707
I20240501 13:59:57.415266 24709 tablet_copy_service.cc:433] P 0e31c1e6d8f843558a29d08245b4c93d: ending tablet copy session 68eab98270d449dbbccecfc3a4cf8766-21494c260afc4690a2269129f4c4582f on tablet 21494c260afc4690a2269129f4c4582f with peer 68eab98270d449dbbccecfc3a4cf8766
I20240501 13:59:57.783592 28439 raft_consensus.cc:1203] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:57.785387 28599 raft_consensus.cc:2932] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:57.786101 28439 raft_consensus.cc:2932] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:57.789353 28599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49868 (request call id 0) took 1032 ms. Trace:
I20240501 13:59:57.789577 28599 rpcz_store.cc:276] 0501 13:59:56.756783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:56.756857 (+ 74us) service_pool.cc:225] Handling call
0501 13:59:56.758139 (+ 1282us) log.cc:842] Serialized 257 byte log entry
0501 13:59:57.789302 (+1031163us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":75,"raft.queue_time_us":395,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":268,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":342,"wal-append.run_cpu_time_us":545,"wal-append.run_wall_time_us":1000606}
I20240501 13:59:57.792179 24546 catalog_manager.cc:5484] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d reported cstate change: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "0e31c1e6d8f843558a29d08245b4c93d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:57.822799 24820 consensus_queue.cc:237] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
W20240501 13:59:57.830473 24756 consensus_peers.cc:485] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 -> Peer 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72:40329): Couldn't send request to peer 68eab98270d449dbbccecfc3a4cf8766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b5f4f1d9e31406ca7bbebf02decc909. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:58.425369 28628 ts_tablet_manager.cc:902] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Initiating tablet copy from peer 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:58.427242 28628 tablet_copy_client.cc:308] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:40637
I20240501 13:59:58.434967 24840 tablet_copy_service.cc:140] P 85c2edb136794ec7a6e4352c57090d79: Received BeginTabletCopySession request for tablet 7b5f4f1d9e31406ca7bbebf02decc909 from peer 68eab98270d449dbbccecfc3a4cf8766 ({username='slave'} at 127.22.33.72:37791)
I20240501 13:59:58.435142 24840 tablet_copy_service.cc:161] P 85c2edb136794ec7a6e4352c57090d79: Beginning new tablet copy session on tablet 7b5f4f1d9e31406ca7bbebf02decc909 from peer 68eab98270d449dbbccecfc3a4cf8766 at {username='slave'} at 127.22.33.72:37791: session id = 68eab98270d449dbbccecfc3a4cf8766-7b5f4f1d9e31406ca7bbebf02decc909
I20240501 13:59:58.437111 24840 tablet_copy_source_session.cc:205] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:58.438362 28628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5f4f1d9e31406ca7bbebf02decc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:58.442948 28628 tablet_copy_client.cc:783] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:58.443205 28628 tablet_copy_client.cc:647] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:58.445111 28628 tablet_copy_client.cc:523] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:58.447551 28628 tablet_bootstrap.cc:492] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap starting.
I20240501 13:59:58.478381 28628 tablet_bootstrap.cc:492] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 13:59:58.478778 28628 tablet_bootstrap.cc:492] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Bootstrap complete.
I20240501 13:59:58.479028 28628 ts_tablet_manager.cc:1367] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.005s
I20240501 13:59:58.479841 28628 raft_consensus.cc:348] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } }
I20240501 13:59:58.480229 28628 raft_consensus.cc:729] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eab98270d449dbbccecfc3a4cf8766, State: Initialized, Role: FOLLOWER
I20240501 13:59:58.480394 28628 consensus_queue.cc:260] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:58.481112 28628 ts_tablet_manager.cc:1398] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:58.481942 24840 tablet_copy_service.cc:342] P 85c2edb136794ec7a6e4352c57090d79: Request end of tablet copy session 68eab98270d449dbbccecfc3a4cf8766-7b5f4f1d9e31406ca7bbebf02decc909 received from {username='slave'} at 127.22.33.72:37791
I20240501 13:59:58.482079 24840 tablet_copy_service.cc:433] P 85c2edb136794ec7a6e4352c57090d79: ending tablet copy session 68eab98270d449dbbccecfc3a4cf8766-7b5f4f1d9e31406ca7bbebf02decc909 on tablet 7b5f4f1d9e31406ca7bbebf02decc909 with peer 68eab98270d449dbbccecfc3a4cf8766
I20240501 13:59:58.936610 28439 raft_consensus.cc:1203] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:58.938059 28638 raft_consensus.cc:2932] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:58.938689 28439 raft_consensus.cc:2932] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:58.941526 28638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36090 (request call id 0) took 1119 ms. Trace:
I20240501 13:59:58.941697 28638 rpcz_store.cc:276] 0501 13:59:57.822239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:59:57.822313 (+ 74us) service_pool.cc:225] Handling call
0501 13:59:57.823445 (+ 1132us) log.cc:842] Serialized 257 byte log entry
0501 13:59:58.941196 (+1117751us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":63,"raft.queue_time_us":380,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":239,"thread_start_us":515,"threads_started":2,"wal-append.queue_time_us":343,"wal-append.run_cpu_time_us":571,"wal-append.run_wall_time_us":1000644}
I20240501 13:59:58.943320 24543 catalog_manager.cc:5484] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 reported cstate change: config changed from index -1 to 4, VOTER 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "85c2edb136794ec7a6e4352c57090d79" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:59.047648 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.061816 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.070416 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.084264 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.103791 28419 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:59.105129 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.154814 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.159600 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.212664 24820 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b5f4f1d9e31406ca7bbebf02decc909"
dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36102
I20240501 13:59:59.212880 24820 raft_consensus.cc:595] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:59.311444 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.325171 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.338421 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.340497 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.345302 28419 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:59.355424 24800 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.367074 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.386755 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.432809 28638 raft_consensus.cc:979] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: : Instructing follower 68eab98270d449dbbccecfc3a4cf8766 to start an election
I20240501 13:59:59.433054 28638 raft_consensus.cc:1067] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Signalling peer 68eab98270d449dbbccecfc3a4cf8766 to start an election
I20240501 13:59:59.433830 28439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b5f4f1d9e31406ca7bbebf02decc909"
dest_uuid: "68eab98270d449dbbccecfc3a4cf8766"
from {username='slave'} at 127.22.33.66:49269
I20240501 13:59:59.434140 28439 raft_consensus.cc:482] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:59.434371 28439 raft_consensus.cc:3037] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:59.436592 28439 raft_consensus.cc:504] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.438064 28439 leader_election.cc:288] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [CANDIDATE]: Term 2 election: Requested vote from peers 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637)
I20240501 13:59:59.438661 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5f4f1d9e31406ca7bbebf02decc909" candidate_uuid: "68eab98270d449dbbccecfc3a4cf8766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85c2edb136794ec7a6e4352c57090d79"
I20240501 13:59:59.438892 24820 raft_consensus.cc:3032] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:59.439002 24820 raft_consensus.cc:729] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85c2edb136794ec7a6e4352c57090d79, State: Running, Role: LEADER
I20240501 13:59:59.439291 24820 consensus_queue.cc:260] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.439584 24820 raft_consensus.cc:3037] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:59.441937 24820 raft_consensus.cc:2454] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eab98270d449dbbccecfc3a4cf8766 in term 2.
I20240501 13:59:59.442747 28373 leader_election.cc:302] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68eab98270d449dbbccecfc3a4cf8766, 85c2edb136794ec7a6e4352c57090d79; no voters:
I20240501 13:59:59.444108 28729 raft_consensus.cc:2781] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:59.444609 28729 raft_consensus.cc:686] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 2 LEADER]: Becoming Leader. State: Replica: 68eab98270d449dbbccecfc3a4cf8766, State: Running, Role: LEADER
I20240501 13:59:59.445191 28729 consensus_queue.cc:237] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.448302 24545 catalog_manager.cc:5484] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 reported cstate change: term changed from 1 to 2, leader changed from 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) to 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72). New cstate: current_term: 2 leader_uuid: "68eab98270d449dbbccecfc3a4cf8766" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 13:59:59.551489 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.564244 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.567478 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.571902 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.572863 28419 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.581499 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.583521 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.603713 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.678453 24689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21494c260afc4690a2269129f4c4582f"
dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39770
I20240501 13:59:59.678691 24689 raft_consensus.cc:595] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:59.766947 24669 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:59.782735 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.783707 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.786660 28419 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.792457 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.820674 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.846719 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.854595 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.905874 24820 raft_consensus.cc:1261] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Refusing update from remote peer 68eab98270d449dbbccecfc3a4cf8766: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 13:59:59.906817 28729 consensus_queue.cc:1035] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c2edb136794ec7a6e4352c57090d79" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:59.931753 28808 raft_consensus.cc:979] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: : Instructing follower 68eab98270d449dbbccecfc3a4cf8766 to start an election
I20240501 13:59:59.931996 28599 raft_consensus.cc:1067] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Signalling peer 68eab98270d449dbbccecfc3a4cf8766 to start an election
I20240501 13:59:59.932755 28439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21494c260afc4690a2269129f4c4582f"
dest_uuid: "68eab98270d449dbbccecfc3a4cf8766"
from {username='slave'} at 127.22.33.65:60207
I20240501 13:59:59.933084 28439 raft_consensus.cc:482] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:59.933207 28439 raft_consensus.cc:3037] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:59.937211 28439 raft_consensus.cc:504] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.937969 28439 leader_election.cc:288] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [CANDIDATE]: Term 2 election: Requested vote from peers 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237)
I20240501 13:59:59.938709 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21494c260afc4690a2269129f4c4582f" candidate_uuid: "68eab98270d449dbbccecfc3a4cf8766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0e31c1e6d8f843558a29d08245b4c93d"
I20240501 13:59:59.938948 24689 raft_consensus.cc:3032] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:59.939023 24689 raft_consensus.cc:729] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e31c1e6d8f843558a29d08245b4c93d, State: Running, Role: LEADER
I20240501 13:59:59.939291 24689 consensus_queue.cc:260] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.939579 24689 raft_consensus.cc:3037] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:59.942426 24689 raft_consensus.cc:2454] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eab98270d449dbbccecfc3a4cf8766 in term 2.
I20240501 13:59:59.943126 28372 leader_election.cc:302] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e31c1e6d8f843558a29d08245b4c93d, 68eab98270d449dbbccecfc3a4cf8766; no voters:
I20240501 13:59:59.943533 28730 raft_consensus.cc:2781] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:59.945466 28730 raft_consensus.cc:686] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 2 LEADER]: Becoming Leader. State: Replica: 68eab98270d449dbbccecfc3a4cf8766, State: Running, Role: LEADER
I20240501 13:59:59.945808 28730 consensus_queue.cc:237] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 13:59:59.949904 24546 catalog_manager.cc:5484] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 reported cstate change: term changed from 1 to 2, leader changed from 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) to 68eab98270d449dbbccecfc3a4cf8766 (127.22.33.72). New cstate: current_term: 2 leader_uuid: "68eab98270d449dbbccecfc3a4cf8766" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 } } peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 14:00:00.306528 28419 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.310587 24669 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.310997 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.321651 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.323386 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.337533 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.338833 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.365082 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.452600 28439 consensus_queue.cc:237] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 14:00:00.453541 28729 raft_consensus.cc:2932] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 14:00:00.458117 24544 catalog_manager.cc:5484] T 7b5f4f1d9e31406ca7bbebf02decc909 P 68eab98270d449dbbccecfc3a4cf8766 reported cstate change: config changed from index 4 to 6, VOTER 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66) evicted. New cstate: current_term: 2 leader_uuid: "68eab98270d449dbbccecfc3a4cf8766" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 14:00:00.463557 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5f4f1d9e31406ca7bbebf02decc909 with delete_type TABLET_DATA_TOMBSTONED (TS 85c2edb136794ec7a6e4352c57090d79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39914
I20240501 14:00:00.464232 28884 tablet_replica.cc:330] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: stopping tablet replica
I20240501 14:00:00.464408 28884 raft_consensus.cc:2229] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:00.464622 28884 raft_consensus.cc:2258] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:00.466701 28884 ts_tablet_manager.cc:1875] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:00.473719 28884 ts_tablet_manager.cc:1888] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:00:00.473877 28884 log.cc:1196] T 7b5f4f1d9e31406ca7bbebf02decc909 P 85c2edb136794ec7a6e4352c57090d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b5f4f1d9e31406ca7bbebf02decc909
I20240501 14:00:00.474777 24533 catalog_manager.cc:4830] TS 85c2edb136794ec7a6e4352c57090d79 (127.22.33.66:40637): tablet 7b5f4f1d9e31406ca7bbebf02decc909 (table rebalance_test_table_3 [id=6aa5be64291c4437be7522d23815ff63]) successfully deleted
I20240501 14:00:00.563001 24689 raft_consensus.cc:1261] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Refusing update from remote peer 68eab98270d449dbbccecfc3a4cf8766: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:00:00.563726 28729 consensus_queue.cc:1035] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e31c1e6d8f843558a29d08245b4c93d" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:00.572737 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.589874 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.592154 28419 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.596724 24669 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.597587 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.641564 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.650285 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.662992 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:00.751469 28439 consensus_queue.cc:237] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } }
I20240501 14:00:00.752367 28729 raft_consensus.cc:2932] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 14:00:00.756614 24546 catalog_manager.cc:5484] T 21494c260afc4690a2269129f4c4582f P 68eab98270d449dbbccecfc3a4cf8766 reported cstate change: config changed from index 4 to 6, VOTER 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "68eab98270d449dbbccecfc3a4cf8766" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68eab98270d449dbbccecfc3a4cf8766" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 40329 } } }
I20240501 14:00:00.760350 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet 21494c260afc4690a2269129f4c4582f with delete_type TABLET_DATA_TOMBSTONED (TS 0e31c1e6d8f843558a29d08245b4c93d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37824
I20240501 14:00:00.761106 28939 tablet_replica.cc:330] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: stopping tablet replica
I20240501 14:00:00.761287 28939 raft_consensus.cc:2229] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:00.761451 28939 raft_consensus.cc:2258] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:00.763033 28939 ts_tablet_manager.cc:1875] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:00.767635 28939 ts_tablet_manager.cc:1888] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:00:00.767765 28939 log.cc:1196] T 21494c260afc4690a2269129f4c4582f P 0e31c1e6d8f843558a29d08245b4c93d: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/21494c260afc4690a2269129f4c4582f
I20240501 14:00:00.768574 24532 catalog_manager.cc:4830] TS 0e31c1e6d8f843558a29d08245b4c93d (127.22.33.65:36237): tablet 21494c260afc4690a2269129f4c4582f (table rebalance_test_table_4 [id=3159197c76384db986f1a1d365f5c298]) successfully deleted
I20240501 14:00:01.077639 24669 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.087271 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.096298 28419 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.100479 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.107769 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.110339 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.114748 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.154439 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.312319 24669 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.315399 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.315759 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.322576 28419 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.348457 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.348969 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.366295 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.375275 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.689354 24669 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.706746 28419 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.714195 25894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.717847 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.729624 26035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.733309 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.744171 24800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.757702 26165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ad3ae472198240bf908bfe32012c5a39 | 127.22.33.126:41135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.33.84:41329 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0e31c1e6d8f843558a29d08245b4c93d | 127.22.33.65:36237 | HEALTHY | <none> | 2 | 0
2c3d9ee3986341868e534aef0a98e8b0 | 127.22.33.71:37917 | HEALTHY | <none> | 2 | 0
35f855d8b43040f7843747f312c33239 | 127.22.33.70:37695 | HEALTHY | <none> | 2 | 0
51db4831ba4348c3a502b54ae65eed48 | 127.22.33.68:44081 | HEALTHY | <none> | 2 | 0
68eab98270d449dbbccecfc3a4cf8766 | 127.22.33.72:40329 | HEALTHY | <none> | 2 | 0
85c2edb136794ec7a6e4352c57090d79 | 127.22.33.66:40637 | HEALTHY | <none> | 2 | 0
ae9b30f780af481cbacd0566ab93e248 | 127.22.33.69:34503 | HEALTHY | <none> | 2 | 0
b481b70e6d1f41408327ab1de26d7406 | 127.22.33.67:37985 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.33.65 | experimental | 127.22.33.65:36237
local_ip_for_outbound_sockets | 127.22.33.66 | experimental | 127.22.33.66:40637
local_ip_for_outbound_sockets | 127.22.33.67 | experimental | 127.22.33.67:37985
local_ip_for_outbound_sockets | 127.22.33.68 | experimental | 127.22.33.68:44081
local_ip_for_outbound_sockets | 127.22.33.69 | experimental | 127.22.33.69:34503
local_ip_for_outbound_sockets | 127.22.33.70 | experimental | 127.22.33.70:37695
local_ip_for_outbound_sockets | 127.22.33.71 | experimental | 127.22.33.71:37917
local_ip_for_outbound_sockets | 127.22.33.72 | experimental | 127.22.33.72:40329
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.33.65:36237
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.33.66:40637
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.33.67:37985
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.33.68:44081
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.33.69:34503
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.33.70:37695
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.33.71:37917
server_dump_info_path | /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.33.72:40329
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.33.84:41329 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:00:01.831959 22661 log_verifier.cc:126] Checking tablet 04a8ca79aa05482c898141739d0ebd7d
I20240501 14:00:01.841526 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04a8ca79aa05482c898141739d0ebd7d
I20240501 14:00:01.841624 22661 log_verifier.cc:126] Checking tablet 21494c260afc4690a2269129f4c4582f
I20240501 14:00:01.854025 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 21494c260afc4690a2269129f4c4582f
I20240501 14:00:01.854168 22661 log_verifier.cc:126] Checking tablet 333670a3a08c4ef4b84def2560eda9c0
I20240501 14:00:01.864049 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 333670a3a08c4ef4b84def2560eda9c0
I20240501 14:00:01.864151 22661 log_verifier.cc:126] Checking tablet 50555d4414914565859e6c59897edb0c
I20240501 14:00:01.874186 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 50555d4414914565859e6c59897edb0c
I20240501 14:00:01.874276 22661 log_verifier.cc:126] Checking tablet 5e87402f3d5d42e5a5df4cccc16a20b2
I20240501 14:00:01.879292 22661 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5e87402f3d5d42e5a5df4cccc16a20b2
I20240501 14:00:01.879377 22661 log_verifier.cc:126] Checking tablet 6548d5ccc1454e53824d414b09e7e86e
I20240501 14:00:01.889120 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6548d5ccc1454e53824d414b09e7e86e
I20240501 14:00:01.889207 22661 log_verifier.cc:126] Checking tablet 7b5f4f1d9e31406ca7bbebf02decc909
I20240501 14:00:01.901382 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7b5f4f1d9e31406ca7bbebf02decc909
I20240501 14:00:01.901485 22661 log_verifier.cc:126] Checking tablet 840930a628ee422792b0c35c0278497a
I20240501 14:00:01.911152 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 840930a628ee422792b0c35c0278497a
I20240501 14:00:01.911238 22661 log_verifier.cc:126] Checking tablet 9f89ef680f644d51baf2fd632a429af7
I20240501 14:00:01.920246 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9f89ef680f644d51baf2fd632a429af7
I20240501 14:00:01.920361 22661 log_verifier.cc:126] Checking tablet b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 14:00:01.934621 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b35dc4ce0bf144d19068653fdfd1e1b8
I20240501 14:00:01.934737 22661 log_verifier.cc:126] Checking tablet bb1d595a67cb40e195e9db6841978c0d
I20240501 14:00:01.949534 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bb1d595a67cb40e195e9db6841978c0d
I20240501 14:00:01.949637 22661 log_verifier.cc:126] Checking tablet c406608014684fc9adbdd5566d27b13a
I20240501 14:00:01.959646 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c406608014684fc9adbdd5566d27b13a
I20240501 14:00:01.959740 22661 log_verifier.cc:126] Checking tablet c757d90176b14f81b9804e2eab04c658
I20240501 14:00:01.969373 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c757d90176b14f81b9804e2eab04c658
I20240501 14:00:01.969460 22661 log_verifier.cc:126] Checking tablet c82ba04565984d99967e1841826acce2
I20240501 14:00:01.979319 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c82ba04565984d99967e1841826acce2
I20240501 14:00:01.979405 22661 log_verifier.cc:126] Checking tablet c8caad758e6243e48008fe469425fcff
I20240501 14:00:01.988545 22661 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c8caad758e6243e48008fe469425fcff
I20240501 14:00:01.988636 22661 log_verifier.cc:126] Checking tablet f7c2617355e1470fbd3a29c59830a8e6
I20240501 14:00:01.998543 22661 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f7c2617355e1470fbd3a29c59830a8e6
I20240501 14:00:01.999859 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24607
I20240501 14:00:02.001498 24729 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.204492 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24607
I20240501 14:00:02.238238 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24738
I20240501 14:00:02.239257 24860 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.438344 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24738
I20240501 14:00:02.471091 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 25831
I20240501 14:00:02.477394 25954 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.653858 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 25831
I20240501 14:00:02.683406 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 25969
I20240501 14:00:02.689498 26094 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.867784 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 25969
I20240501 14:00:02.896533 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 26103
I20240501 14:00:02.902463 26224 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.082208 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 26103
I20240501 14:00:03.111598 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 26233
I20240501 14:00:03.118069 26354 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.297103 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 26233
I20240501 14:00:03.326201 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 26363
I20240501 14:00:03.332234 26484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.510984 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 26363
I20240501 14:00:03.540251 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 28357
I20240501 14:00:03.546499 28478 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.658244 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 28357
I20240501 14:00:03.678750 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 24517
I20240501 14:00:03.679805 24578 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.905591 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 24517
2024-05-01T14:00:03Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (44519 ms)
[----------] 1 test from RebalanceParamTest (44519 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 14:00:03.987339 22661 test_util.cc:268] Using random seed: -415412367
I20240501 14:00:03.989257 22661 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:00:03.989317 22661 ts_itest-base.cc:113] --------------
I20240501 14:00:03.989357 22661 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:00:03.989393 22661 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:00:03.989435 22661 ts_itest-base.cc:116] --------------
2024-05-01T14:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:00:04Z Disabled control of system clock
I20240501 14:00:04.019061 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:35223
--webserver_interface=127.22.33.126
--webserver_port=0
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.33.126:35223
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:00:04.353391 29074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:04.353724 29074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:04.353853 29074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:04.361347 29074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:04.361477 29074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:00:04.361536 29074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:04.361619 29074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:00:04.361681 29074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:00:04.370836 29074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.33.126:35223
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.33.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:04.372265 29074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:04.374025 29074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:04.385357 29079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.385947 29080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.386593 29082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:04.387351 29074 server_base.cc:1034] running on GCE node
I20240501 14:00:04.388252 29074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:04.389763 29074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:04.390969 29074 hybrid_clock.cc:648] HybridClock initialized: now 1714572004390906 us; error 59 us; skew 500 ppm
I20240501 14:00:04.391335 29074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:04.393680 29074 webserver.cc:460] Webserver started at http://127.22.33.126:34999/ using document root <none> and password file <none>
I20240501 14:00:04.394426 29074 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:04.394527 29074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:04.394825 29074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:04.397505 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "589b051e6abe4fa38c771a41c6df7284"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.398262 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "589b051e6abe4fa38c771a41c6df7284"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.403267 29074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:00:04.406463 29088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:04.407914 29074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:04.408067 29074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
uuid: "589b051e6abe4fa38c771a41c6df7284"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.408272 29074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:04.434334 29074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:04.435240 29074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:04.435506 29074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:04.454097 29074 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.126:35223
I20240501 14:00:04.454124 29139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.126:35223 every 8 connection(s)
I20240501 14:00:04.455511 29074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:00:04.459314 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29074
I20240501 14:00:04.459340 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.459523 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:00:04.473881 29140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Bootstrap starting.
I20240501 14:00:04.477320 29140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:04.478411 29140 log.cc:824] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:04.481158 29140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: No bootstrap required, opened a new log
I20240501 14:00:04.488466 29140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } }
I20240501 14:00:04.488878 29140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:04.488996 29140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589b051e6abe4fa38c771a41c6df7284, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.489631 29140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } }
I20240501 14:00:04.489852 29140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:04.489969 29140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:04.490123 29140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.492496 29140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } }
I20240501 14:00:04.493127 29140 leader_election.cc:302] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589b051e6abe4fa38c771a41c6df7284; no voters:
I20240501 14:00:04.493927 29140 leader_election.cc:288] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:04.494027 29145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:04.494834 29145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [term 1 LEADER]: Becoming Leader. State: Replica: 589b051e6abe4fa38c771a41c6df7284, State: Running, Role: LEADER
I20240501 14:00:04.495406 29145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } }
I20240501 14:00:04.496196 29140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:04.498332 29146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589b051e6abe4fa38c771a41c6df7284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } } }
I20240501 14:00:04.498769 29146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:04.499532 29147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589b051e6abe4fa38c771a41c6df7284. Latest consensus state: current_term: 1 leader_uuid: "589b051e6abe4fa38c771a41c6df7284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589b051e6abe4fa38c771a41c6df7284" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 35223 } } }
I20240501 14:00:04.499922 29147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:04.501649 29153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:04.508459 29153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:04.517508 29153 catalog_manager.cc:1347] Generated new cluster ID: da89eeeabe2a43f4a4c60f1169f8e4da
I20240501 14:00:04.517645 29153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:04.530580 29153 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:04.531841 29153 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:04.539721 29153 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Generated new TSK 0
I20240501 14:00:04.540727 29153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:04.559970 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.65:0
--local_ip_for_outbound_sockets=127.22.33.65
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:00:04.898576 29164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:04.898866 29164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:04.898954 29164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:04.907263 29164 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:04.907390 29164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:04.907509 29164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.65
I20240501 14:00:04.916836 29164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.65
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:04.918293 29164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:04.920034 29164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:04.932262 29172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.932996 29170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.933593 29169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:04.934474 29164 server_base.cc:1034] running on GCE node
I20240501 14:00:04.935094 29164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:04.936017 29164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:04.937213 29164 hybrid_clock.cc:648] HybridClock initialized: now 1714572004937158 us; error 48 us; skew 500 ppm
I20240501 14:00:04.937515 29164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:04.939934 29164 webserver.cc:460] Webserver started at http://127.22.33.65:40385/ using document root <none> and password file <none>
I20240501 14:00:04.940685 29164 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:04.940800 29164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:04.941159 29164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:04.945663 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.946450 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.951591 29164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240501 14:00:04.954937 29178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:04.956254 29164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:04.956408 29164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-3610"
I20240501 14:00:04.956596 29164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:04.971629 29164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:04.972462 29164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:04.972733 29164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:04.973749 29164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:04.975284 29164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:04.975387 29164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:04.975507 29164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:04.975579 29164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.006759 29164 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.65:43201
I20240501 14:00:05.006814 29290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.65:43201 every 8 connection(s)
I20240501 14:00:05.008051 29164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:00:05.015765 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29164
I20240501 14:00:05.015960 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:00:05.019382 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.66:0
--local_ip_for_outbound_sockets=127.22.33.66
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:05.020560 29291 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:05.020890 29291 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:05.021798 29291 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:05.023847 29105 ts_manager.cc:194] Registered new tserver with Master: 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:05.025324 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.65:58749
W20240501 14:00:05.350883 29295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:05.351166 29295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:05.351311 29295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:05.358685 29295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:05.358811 29295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:05.358992 29295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.66
I20240501 14:00:05.367908 29295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.66
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:05.369258 29295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:05.370939 29295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:05.383347 29300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.383769 29301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.384585 29303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:05.385447 29295 server_base.cc:1034] running on GCE node
I20240501 14:00:05.386025 29295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:05.387008 29295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:05.388217 29295 hybrid_clock.cc:648] HybridClock initialized: now 1714572005388169 us; error 41 us; skew 500 ppm
I20240501 14:00:05.388537 29295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:05.390797 29295 webserver.cc:460] Webserver started at http://127.22.33.66:37261/ using document root <none> and password file <none>
I20240501 14:00:05.391533 29295 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:05.391649 29295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:05.391973 29295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:05.394583 29295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b87d188552494b4590afc37a85b98c35"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.395361 29295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b87d188552494b4590afc37a85b98c35"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.400579 29295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:00:05.404084 29309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.405459 29295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:05.405606 29295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b87d188552494b4590afc37a85b98c35"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.405791 29295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:05.428464 29295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:05.429349 29295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:05.429610 29295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:05.430594 29295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:05.432077 29295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:05.432191 29295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.432309 29295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:05.432394 29295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.464658 29295 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.66:39595
I20240501 14:00:05.464741 29421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.66:39595 every 8 connection(s)
I20240501 14:00:05.465998 29295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:00:05.470813 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29295
I20240501 14:00:05.471002 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:00:05.473963 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:0
--local_ip_for_outbound_sockets=127.22.33.67
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:05.479648 29422 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:05.479957 29422 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:05.480756 29422 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:05.482071 29105 ts_manager.cc:194] Registered new tserver with Master: b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:05.482789 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.66:43589
W20240501 14:00:05.810196 29425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:05.810544 29425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:05.810703 29425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:05.818365 29425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:05.818490 29425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:05.818694 29425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 14:00:05.827867 29425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:05.829262 29425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:05.830965 29425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:05.843147 29430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.843753 29433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.844429 29431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:05.845129 29425 server_base.cc:1034] running on GCE node
I20240501 14:00:05.845721 29425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:05.846660 29425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:05.847859 29425 hybrid_clock.cc:648] HybridClock initialized: now 1714572005847815 us; error 40 us; skew 500 ppm
I20240501 14:00:05.848173 29425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:05.854476 29425 webserver.cc:460] Webserver started at http://127.22.33.67:35375/ using document root <none> and password file <none>
I20240501 14:00:05.855269 29425 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:05.855404 29425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:05.855728 29425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:05.858438 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "522f4ff5e9084cbf8462196533c7d7aa"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.859256 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "522f4ff5e9084cbf8462196533c7d7aa"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.864635 29425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:00:05.868080 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.869442 29425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:05.869597 29425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "522f4ff5e9084cbf8462196533c7d7aa"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-3610"
I20240501 14:00:05.869789 29425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:05.890455 29425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:05.891343 29425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:05.891600 29425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:05.892536 29425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:05.894032 29425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:05.894125 29425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.894234 29425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:05.894313 29425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.925853 29425 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:44187
I20240501 14:00:05.925905 29551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:44187 every 8 connection(s)
I20240501 14:00:05.927285 29425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:00:05.935002 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29425
I20240501 14:00:05.935204 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:00:05.938397 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:0
--local_ip_for_outbound_sockets=127.22.33.68
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:05.941210 29552 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:05.941537 29552 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:05.942335 29552 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:05.943885 29105 ts_manager.cc:194] Registered new tserver with Master: 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:05.944705 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:50001
I20240501 14:00:06.028538 29291 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
W20240501 14:00:06.280072 29555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:06.280413 29555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:06.280516 29555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:06.288177 29555 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:06.288311 29555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:06.288451 29555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 14:00:06.297829 29555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:06.299168 29555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:06.300962 29555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:06.313777 29563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.314258 29561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.314754 29560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:06.315726 29555 server_base.cc:1034] running on GCE node
I20240501 14:00:06.316399 29555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:06.317487 29555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:06.318732 29555 hybrid_clock.cc:648] HybridClock initialized: now 1714572006318678 us; error 50 us; skew 500 ppm
I20240501 14:00:06.319075 29555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:06.321597 29555 webserver.cc:460] Webserver started at http://127.22.33.68:46421/ using document root <none> and password file <none>
I20240501 14:00:06.322364 29555 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:06.322487 29555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:06.322805 29555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:06.325589 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.326392 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.331955 29555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:00:06.335523 29569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.337031 29555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:06.337205 29555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.337407 29555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:06.362903 29555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:06.363802 29555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:06.364078 29555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:06.365104 29555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:06.366921 29555 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:06.367020 29555 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.367130 29555 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:06.367213 29555 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.398584 29555 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:34947
I20240501 14:00:06.398653 29681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:34947 every 8 connection(s)
I20240501 14:00:06.399907 29555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:00:06.401388 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29555
I20240501 14:00:06.401579 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:00:06.404681 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:0
--local_ip_for_outbound_sockets=127.22.33.69
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:06.413628 29682 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:06.413964 29682 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:06.414680 29682 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:06.416075 29105 ts_manager.cc:194] Registered new tserver with Master: ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:06.416822 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:52573
I20240501 14:00:06.485213 29422 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
W20240501 14:00:06.743840 29685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:06.744139 29685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:06.744243 29685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:06.751803 29685 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:06.751940 29685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:06.752056 29685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 14:00:06.761139 29685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:06.762441 29685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:06.764068 29685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:06.776265 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.776865 29693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.777660 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:06.778352 29685 server_base.cc:1034] running on GCE node
I20240501 14:00:06.778920 29685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:06.779801 29685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:06.781044 29685 hybrid_clock.cc:648] HybridClock initialized: now 1714572006780988 us; error 48 us; skew 500 ppm
I20240501 14:00:06.781342 29685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:06.783489 29685 webserver.cc:460] Webserver started at http://127.22.33.69:34427/ using document root <none> and password file <none>
I20240501 14:00:06.784210 29685 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:06.784329 29685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:06.784611 29685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:06.787137 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9903ed35bca144bcb74e9bf5102117e7"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.787928 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9903ed35bca144bcb74e9bf5102117e7"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.792923 29685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:00:06.796134 29699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.797499 29685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:06.797649 29685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9903ed35bca144bcb74e9bf5102117e7"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:06.797847 29685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:06.812870 29685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:06.813788 29685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:06.814069 29685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:06.815035 29685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:06.816519 29685 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:06.816614 29685 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.816731 29685 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:06.816800 29685 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.850802 29685 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:32783
I20240501 14:00:06.850836 29811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:32783 every 8 connection(s)
I20240501 14:00:06.852258 29685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:06.856767 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29685
I20240501 14:00:06.856997 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:00:06.860711 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:0
--local_ip_for_outbound_sockets=127.22.33.70
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:06.866607 29812 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:06.866971 29812 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:06.867882 29812 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:06.869323 29105 ts_manager.cc:194] Registered new tserver with Master: 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:06.870048 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:46655
I20240501 14:00:06.947883 29552 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
W20240501 14:00:07.223956 29815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:07.224290 29815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:07.224388 29815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:07.231818 29815 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:07.231958 29815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:07.232082 29815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 14:00:07.241308 29815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:07.242667 29815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:07.244408 29815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:07.257073 29821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:07.257917 29820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:07.258360 29823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:07.259145 29815 server_base.cc:1034] running on GCE node
I20240501 14:00:07.259802 29815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:07.260821 29815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:07.262096 29815 hybrid_clock.cc:648] HybridClock initialized: now 1714572007261997 us; error 68 us; skew 500 ppm
I20240501 14:00:07.262483 29815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:07.264997 29815 webserver.cc:460] Webserver started at http://127.22.33.70:43603/ using document root <none> and password file <none>
I20240501 14:00:07.265812 29815 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:07.265937 29815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:07.266268 29815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:07.269146 29815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "0a697f3a1ee544daa7073814ab845d58"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.269953 29815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "0a697f3a1ee544daa7073814ab845d58"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.275655 29815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:00:07.279219 29829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.280629 29815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:07.280861 29815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0a697f3a1ee544daa7073814ab845d58"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.281164 29815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:07.312762 29815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:07.313748 29815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:07.314030 29815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:07.315006 29815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:07.316534 29815 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:07.316632 29815 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.316748 29815 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:07.316819 29815 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.349406 29815 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:37745
I20240501 14:00:07.349478 29941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:37745 every 8 connection(s)
I20240501 14:00:07.350786 29815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:00:07.353292 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29815
I20240501 14:00:07.353461 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:00:07.356487 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:0
--local_ip_for_outbound_sockets=127.22.33.71
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:07.364166 29942 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:07.364550 29942 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:07.365561 29942 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:07.367251 29105 ts_manager.cc:194] Registered new tserver with Master: 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:07.368062 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:51215
I20240501 14:00:07.419397 29682 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
W20240501 14:00:07.697444 29945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:07.697733 29945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:07.697865 29945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:07.705360 29945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:07.705478 29945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:07.705631 29945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 14:00:07.715303 29945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:07.716663 29945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:07.718457 29945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:07.730901 29953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:07.731563 29951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:07.732228 29950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:07.733156 29945 server_base.cc:1034] running on GCE node
I20240501 14:00:07.733767 29945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:07.734722 29945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:07.735918 29945 hybrid_clock.cc:648] HybridClock initialized: now 1714572007735864 us; error 54 us; skew 500 ppm
I20240501 14:00:07.736231 29945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:07.738515 29945 webserver.cc:460] Webserver started at http://127.22.33.71:34479/ using document root <none> and password file <none>
I20240501 14:00:07.739269 29945 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:07.739382 29945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:07.739679 29945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:07.742408 29945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6a446ac802a7457d9f27076781f21d79"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.743165 29945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6a446ac802a7457d9f27076781f21d79"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.748353 29945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 14:00:07.751833 29959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.753252 29945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:00:07.753413 29945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6a446ac802a7457d9f27076781f21d79"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:07.753607 29945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:07.770470 29945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:07.771404 29945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:07.771724 29945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:07.772794 29945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:07.774451 29945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:07.774540 29945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.774648 29945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:07.774717 29945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.806352 29945 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:38743
I20240501 14:00:07.806394 30071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:38743 every 8 connection(s)
I20240501 14:00:07.807652 29945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:00:07.808895 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 29945
I20240501 14:00:07.809136 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:00:07.812397 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.72:0
--local_ip_for_outbound_sockets=127.22.33.72
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:07.820924 30072 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:07.821298 30072 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:07.822062 30072 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:07.823438 29105 ts_manager.cc:194] Registered new tserver with Master: 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:07.824226 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:51855
I20240501 14:00:07.873144 29812 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
W20240501 14:00:08.147421 30075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:08.147754 30075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:08.147892 30075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:08.155612 30075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:08.155771 30075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:08.155951 30075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.72
I20240501 14:00:08.165565 30075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.72
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:08.166922 30075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:08.168607 30075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:08.181160 30080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.181748 30083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.182368 30081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.183136 30075 server_base.cc:1034] running on GCE node
I20240501 14:00:08.183686 30075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:08.184631 30075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:08.185822 30075 hybrid_clock.cc:648] HybridClock initialized: now 1714572008185744 us; error 69 us; skew 500 ppm
I20240501 14:00:08.186184 30075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:08.188555 30075 webserver.cc:460] Webserver started at http://127.22.33.72:44653/ using document root <none> and password file <none>
I20240501 14:00:08.189338 30075 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:08.189448 30075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:08.189749 30075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:08.192333 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6edd6a3430d44c38a19c2804e555059f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-3610"
I20240501 14:00:08.193112 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6edd6a3430d44c38a19c2804e555059f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-3610"
I20240501 14:00:08.198264 30075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:08.201591 30089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.203028 30075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:08.203204 30075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6edd6a3430d44c38a19c2804e555059f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-3610"
I20240501 14:00:08.203408 30075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:08.233606 30075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:08.234544 30075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:08.234839 30075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:08.235842 30075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:08.237556 30075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:08.237653 30075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.237752 30075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:08.237813 30075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.267652 30075 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.72:37615
I20240501 14:00:08.267710 30201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.72:37615 every 8 connection(s)
I20240501 14:00:08.268949 30075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:00:08.273797 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 30075
I20240501 14:00:08.274027 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:00:08.282125 30202 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:08.282395 30202 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:08.283039 30202 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:08.284242 29105 ts_manager.cc:194] Registered new tserver with Master: 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:08.284956 29105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.72:51173
I20240501 14:00:08.290256 22661 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:00:08.306311 22661 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:00:08.306458 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29685
I20240501 14:00:08.314760 29807 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:08.371505 29942 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
I20240501 14:00:08.395871 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29685
I20240501 14:00:08.412528 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29815
I20240501 14:00:08.421183 29937 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:08.505833 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29815
I20240501 14:00:08.522502 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29945
I20240501 14:00:08.530725 30067 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:08.611871 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29945
I20240501 14:00:08.628049 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 30075
I20240501 14:00:08.636409 30197 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:08.717299 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 30075
I20240501 14:00:12.485771 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:12.538139 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.538071 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.539023 29486 tablet_service.cc:1467] Processing CreateTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.538080 29225 tablet_service.cc:1467] Processing CreateTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.539744 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ce76b9ac34f69bb79049abecfb674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.539631 29224 tablet_service.cc:1467] Processing CreateTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.540030 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201cef3c1164bdeb443b3f75ffcb076. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.540197 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ff1d95887a4e2c9b1b2ff84fb45089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.544540 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ce76b9ac34f69bb79049abecfb674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.544889 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ff1d95887a4e2c9b1b2ff84fb45089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.546228 29616 tablet_service.cc:1467] Processing CreateTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.548105 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ff1d95887a4e2c9b1b2ff84fb45089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.547106 29617 tablet_service.cc:1467] Processing CreateTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.549021 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201cef3c1164bdeb443b3f75ffcb076. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.552445 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.553552 29356 tablet_service.cc:1467] Processing CreateTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 (DEFAULT_TABLE table=rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:12.554071 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ce76b9ac34f69bb79049abecfb674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.554589 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201cef3c1164bdeb443b3f75ffcb076. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:12.562963 30232 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:12.565862 30232 tablet_bootstrap.cc:654] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.567095 30232 log.cc:824] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:12.570693 30232 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:12.570992 30232 ts_tablet_manager.cc:1367] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:00:12.602149 30231 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:12.604063 30235 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:12.597363 30232 raft_consensus.cc:348] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.605067 30232 raft_consensus.cc:374] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.605293 30232 raft_consensus.cc:729] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.606478 30232 consensus_queue.cc:260] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.606725 30231 tablet_bootstrap.cc:654] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.607615 30235 tablet_bootstrap.cc:654] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.608088 30231 log.cc:824] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:12.609824 30234 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:12.608954 30235 log.cc:824] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:12.611219 30232 ts_tablet_manager.cc:1398] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.040s user 0.013s sys 0.003s
I20240501 14:00:12.611888 30232 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:12.613543 30234 tablet_bootstrap.cc:654] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.614277 30232 tablet_bootstrap.cc:654] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.614987 30234 log.cc:824] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:12.622149 30231 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:12.622680 30231 ts_tablet_manager.cc:1367] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20240501 14:00:12.631639 30232 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:12.631944 30232 ts_tablet_manager.cc:1367] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.016s
I20240501 14:00:12.631999 30235 tablet_bootstrap.cc:492] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:12.632412 30235 ts_tablet_manager.cc:1367] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.015s
I20240501 14:00:12.633028 30234 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:12.633371 30234 ts_tablet_manager.cc:1367] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.016s
I20240501 14:00:12.634049 30232 raft_consensus.cc:348] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.634506 30232 raft_consensus.cc:374] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.634652 30232 raft_consensus.cc:729] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.634528 30231 raft_consensus.cc:348] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.635015 30231 raft_consensus.cc:374] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.635152 30231 raft_consensus.cc:729] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.635129 30232 consensus_queue.cc:260] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.635890 30232 ts_tablet_manager.cc:1398] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:12.635991 30231 consensus_queue.cc:260] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.636229 30232 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:12.637986 30232 tablet_bootstrap.cc:654] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.644249 30234 raft_consensus.cc:348] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.644906 30234 raft_consensus.cc:374] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.645097 30234 raft_consensus.cc:729] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.646075 30234 consensus_queue.cc:260] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.650872 30235 raft_consensus.cc:348] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.652849 30235 raft_consensus.cc:374] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.653148 30241 raft_consensus.cc:482] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:12.653347 30231 ts_tablet_manager.cc:1398] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.030s user 0.024s sys 0.004s
I20240501 14:00:12.653499 30241 raft_consensus.cc:504] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.653877 30231 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:12.655716 30231 tablet_bootstrap.cc:654] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.655709 30235 raft_consensus.cc:729] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.656733 30235 consensus_queue.cc:260] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.658252 30241 leader_election.cc:288] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:12.660641 30234 ts_tablet_manager.cc:1398] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.027s user 0.014s sys 0.011s
I20240501 14:00:12.661190 30234 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:12.662138 30235 ts_tablet_manager.cc:1398] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.030s user 0.010s sys 0.006s
I20240501 14:00:12.663524 30234 tablet_bootstrap.cc:654] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.664129 30235 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:12.666332 30235 tablet_bootstrap.cc:654] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:12.669625 30231 tablet_bootstrap.cc:492] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:12.669965 30231 ts_tablet_manager.cc:1367] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240501 14:00:12.670938 30232 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:12.671000 30231 raft_consensus.cc:348] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.671152 30232 ts_tablet_manager.cc:1367] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.004s
I20240501 14:00:12.671200 30231 raft_consensus.cc:374] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.671281 30231 raft_consensus.cc:729] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.671499 30231 consensus_queue.cc:260] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.672039 30231 ts_tablet_manager.cc:1398] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:12.672782 30232 raft_consensus.cc:348] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.673131 30232 raft_consensus.cc:374] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.673228 30232 raft_consensus.cc:729] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.674938 30232 consensus_queue.cc:260] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.676340 30232 ts_tablet_manager.cc:1398] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:12.676489 30235 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:12.676625 30235 ts_tablet_manager.cc:1367] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240501 14:00:12.677628 30235 raft_consensus.cc:348] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.677855 30235 raft_consensus.cc:374] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.677956 30235 raft_consensus.cc:729] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.678184 30235 consensus_queue.cc:260] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.678689 30235 ts_tablet_manager.cc:1398] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:12.681584 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ff1d95887a4e2c9b1b2ff84fb45089" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:12.682085 29637 raft_consensus.cc:2454] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:12.682754 29440 leader_election.cc:302] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:12.683171 30241 raft_consensus.cc:2781] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:12.683337 30241 raft_consensus.cc:482] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:12.683432 30241 raft_consensus.cc:3037] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:12.685004 29553 tablet.cc:2369] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:12.686208 30234 tablet_bootstrap.cc:492] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:12.686620 30241 raft_consensus.cc:504] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.687074 30234 ts_tablet_manager.cc:1367] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.006s
I20240501 14:00:12.687309 30241 leader_election.cc:288] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:12.688035 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ff1d95887a4e2c9b1b2ff84fb45089" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:12.688052 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ff1d95887a4e2c9b1b2ff84fb45089" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:12.688249 29637 raft_consensus.cc:3037] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.688293 30234 raft_consensus.cc:348] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.688522 30234 raft_consensus.cc:374] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:12.688380 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ff1d95887a4e2c9b1b2ff84fb45089" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:12.688623 30234 raft_consensus.cc:729] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:12.688685 29246 raft_consensus.cc:2454] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:12.688848 30234 consensus_queue.cc:260] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.689369 30234 ts_tablet_manager.cc:1398] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:12.690925 29637 raft_consensus.cc:2454] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:12.691305 29440 leader_election.cc:302] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:00:12.691612 30241 raft_consensus.cc:2781] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:12.692087 30241 raft_consensus.cc:686] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:12.692620 30241 consensus_queue.cc:237] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:12.696731 29104 catalog_manager.cc:5484] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
W20240501 14:00:12.723973 29423 tablet.cc:2369] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:12.765130 29292 tablet.cc:2369] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:12.861948 30236 raft_consensus.cc:482] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:12.862190 30236 raft_consensus.cc:504] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.863823 30236 leader_election.cc:288] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:12.872462 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201cef3c1164bdeb443b3f75ffcb076" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:12.872817 29637 raft_consensus.cc:2454] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:12.872778 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201cef3c1164bdeb443b3f75ffcb076" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:12.873415 29377 raft_consensus.cc:2454] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:12.873400 29179 leader_election.cc:302] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:12.873790 30236 raft_consensus.cc:2781] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:12.874022 30236 raft_consensus.cc:482] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:12.874187 30236 raft_consensus.cc:3037] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.876376 30236 raft_consensus.cc:504] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.876993 30236 leader_election.cc:288] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:12.877446 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201cef3c1164bdeb443b3f75ffcb076" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:12.877490 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201cef3c1164bdeb443b3f75ffcb076" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:12.877657 29637 raft_consensus.cc:3037] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.877661 29377 raft_consensus.cc:3037] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.879457 29637 raft_consensus.cc:2454] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:12.879657 29377 raft_consensus.cc:2454] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:12.879845 29179 leader_election.cc:302] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:12.880134 30236 raft_consensus.cc:2781] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:12.880523 30236 raft_consensus.cc:686] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:12.881071 30236 consensus_queue.cc:237] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:12.885166 29104 catalog_manager.cc:5484] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
W20240501 14:00:12.906889 29683 tablet.cc:2369] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:12.977483 30236 raft_consensus.cc:482] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:12.977711 30236 raft_consensus.cc:504] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.978438 30236 leader_election.cc:288] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:12.979033 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ce76b9ac34f69bb79049abecfb674" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:12.979319 29377 raft_consensus.cc:2454] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:12.979804 29181 leader_election.cc:302] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:12.980110 30236 raft_consensus.cc:2781] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:12.980253 30236 raft_consensus.cc:482] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:12.980382 30236 raft_consensus.cc:3037] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.983124 30236 raft_consensus.cc:504] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.983995 30236 leader_election.cc:288] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:12.984304 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ce76b9ac34f69bb79049abecfb674" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:12.984575 29377 raft_consensus.cc:3037] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.985559 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ce76b9ac34f69bb79049abecfb674" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:12.985586 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ce76b9ac34f69bb79049abecfb674" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:12.986021 29506 raft_consensus.cc:3037] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:12.987205 29377 raft_consensus.cc:2454] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:12.987586 29181 leader_election.cc:302] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:12.987856 30236 raft_consensus.cc:2781] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:12.988005 30236 raft_consensus.cc:686] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:12.988253 30236 consensus_queue.cc:237] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:12.988384 29506 raft_consensus.cc:2454] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:12.991014 29104 catalog_manager.cc:5484] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:13.103479 30241 consensus_queue.cc:1035] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:13.112428 29245 raft_consensus.cc:3037] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.115471 30261 consensus_queue.cc:1035] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:13.359756 30257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:13.360082 30257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:13.367822 30257 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:00:13.433789 30236 consensus_queue.cc:1035] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:13.441679 30236 consensus_queue.cc:1035] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:13.513661 29377 raft_consensus.cc:1261] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:13.513696 29506 raft_consensus.cc:1261] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:13.514350 30254 consensus_queue.cc:1035] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:13.514638 30236 consensus_queue.cc:1035] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:13.532765 30263 mvcc.cc:205] Tried to move back new op lower bound from 7022886967340621824 to 7022886965202526208. Current Snapshot: MvccSnapshot[applied={T|T < 7022886967340621824}]
I20240501 14:00:13.550293 30288 mvcc.cc:205] Tried to move back new op lower bound from 7022886967340621824 to 7022886965202526208. Current Snapshot: MvccSnapshot[applied={T|T < 7022886967340621824}]
I20240501 14:00:13.584149 30259 mvcc.cc:205] Tried to move back new op lower bound from 7022886967340621824 to 7022886965202526208. Current Snapshot: MvccSnapshot[applied={T|T < 7022886967340621824}]
INSERT report
rows total: 2000
time total: 247.01 ms
time per row: 0.123505 ms
I20240501 14:00:13.757660 29101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:13.766335 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 21c02994dfef4a10a87b1f47d3ab6e3b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.766392 29486 tablet_service.cc:1467] Processing CreateTablet for tablet 21c02994dfef4a10a87b1f47d3ab6e3b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.766825 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c02994dfef4a10a87b1f47d3ab6e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.766840 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c02994dfef4a10a87b1f47d3ab6e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.766880 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 21c02994dfef4a10a87b1f47d3ab6e3b (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.767258 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c02994dfef4a10a87b1f47d3ab6e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.768494 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 8427dee24ad04247b23b24570c73c92a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.768878 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8427dee24ad04247b23b24570c73c92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.770511 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 578c3f4c5a7c4e2c962b64736195f0bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.770509 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 578c3f4c5a7c4e2c962b64736195f0bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.770917 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578c3f4c5a7c4e2c962b64736195f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.770947 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578c3f4c5a7c4e2c962b64736195f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.773350 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 8427dee24ad04247b23b24570c73c92a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.773712 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8427dee24ad04247b23b24570c73c92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.772850 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 8427dee24ad04247b23b24570c73c92a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.777364 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8427dee24ad04247b23b24570c73c92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.776937 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 578c3f4c5a7c4e2c962b64736195f0bc (DEFAULT_TABLE table=rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.778913 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578c3f4c5a7c4e2c962b64736195f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.786448 30322 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:13.788466 30322 tablet_bootstrap.cc:654] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.791821 30324 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:13.794250 30325 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:13.795071 30324 tablet_bootstrap.cc:654] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.798120 30325 tablet_bootstrap.cc:654] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.798478 30322 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:13.798676 30322 ts_tablet_manager.cc:1367] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:00:13.799793 30322 raft_consensus.cc:348] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.800091 30322 raft_consensus.cc:374] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.800205 30322 raft_consensus.cc:729] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.800575 30322 consensus_queue.cc:260] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.801390 30322 ts_tablet_manager.cc:1398] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:13.801739 30322 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:13.803997 30322 tablet_bootstrap.cc:654] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.804698 30261 raft_consensus.cc:482] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:13.804898 30261 raft_consensus.cc:504] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.805980 30261 leader_election.cc:288] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:13.807376 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
W20240501 14:00:13.808454 29440 leader_election.cc:341] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:13.809979 30324 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:13.810189 30324 ts_tablet_manager.cc:1367] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20240501 14:00:13.811307 30324 raft_consensus.cc:348] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.811579 30324 raft_consensus.cc:374] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.811691 30324 raft_consensus.cc:729] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.812019 30324 consensus_queue.cc:260] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.812814 30324 ts_tablet_manager.cc:1398] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:13.813158 30324 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:13.815363 30324 tablet_bootstrap.cc:654] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.818313 30324 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:13.818480 30324 ts_tablet_manager.cc:1367] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:13.819319 30324 raft_consensus.cc:348] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.819538 30324 raft_consensus.cc:374] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.819612 30322 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:13.819626 30324 raft_consensus.cc:729] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.819783 30322 ts_tablet_manager.cc:1367] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240501 14:00:13.819881 30324 consensus_queue.cc:260] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.820585 30324 ts_tablet_manager.cc:1398] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:13.820875 30324 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:13.821477 30322 raft_consensus.cc:348] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.821712 30322 raft_consensus.cc:374] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.821791 30322 raft_consensus.cc:729] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.822032 30322 consensus_queue.cc:260] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.822486 30324 tablet_bootstrap.cc:654] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.822718 30322 ts_tablet_manager.cc:1398] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:13.823021 30322 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:13.825191 30322 tablet_bootstrap.cc:654] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.827698 30322 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:13.827885 30322 ts_tablet_manager.cc:1367] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:00:13.830720 30322 raft_consensus.cc:348] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.831018 30322 raft_consensus.cc:374] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.831120 30322 raft_consensus.cc:729] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.831442 30322 consensus_queue.cc:260] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.833714 30322 ts_tablet_manager.cc:1398] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:00:13.834681 30324 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:13.834887 30324 ts_tablet_manager.cc:1367] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240501 14:00:13.835817 30324 raft_consensus.cc:348] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.836045 30324 raft_consensus.cc:374] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.836151 30324 raft_consensus.cc:729] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.836393 30324 consensus_queue.cc:260] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.837810 30324 ts_tablet_manager.cc:1398] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:13.841058 30325 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:13.841240 30325 ts_tablet_manager.cc:1367] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.016s
I20240501 14:00:13.841094 30323 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:13.842150 30325 raft_consensus.cc:348] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.842367 30325 raft_consensus.cc:374] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.842381 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:13.842473 30325 raft_consensus.cc:729] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.842607 29376 raft_consensus.cc:2454] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:13.842749 30325 consensus_queue.cc:260] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.843302 30325 ts_tablet_manager.cc:1398] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:13.843526 30323 tablet_bootstrap.cc:654] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.843578 30325 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:13.843806 29442 leader_election.cc:302] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:13.844324 30261 raft_consensus.cc:2781] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:13.844426 30261 raft_consensus.cc:482] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:13.844528 30261 raft_consensus.cc:3037] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.845142 30325 tablet_bootstrap.cc:654] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.847807 30261 raft_consensus.cc:504] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.849004 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:13.849237 29637 raft_consensus.cc:3037] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.849486 30261 leader_election.cc:288] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:13.849840 30325 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:13.850035 30325 ts_tablet_manager.cc:1367] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:00:13.850495 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:13.850709 29376 raft_consensus.cc:3037] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.850931 30325 raft_consensus.cc:348] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.851156 30325 raft_consensus.cc:374] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.851253 30325 raft_consensus.cc:729] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.851516 30325 consensus_queue.cc:260] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:13.852046 30323 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:13.852059 30325 ts_tablet_manager.cc:1398] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:13.852196 29637 raft_consensus.cc:2454] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:13.852350 30323 ts_tablet_manager.cc:1367] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 14:00:13.852684 29440 leader_election.cc:302] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:13.853302 30261 raft_consensus.cc:2781] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:13.853294 30323 raft_consensus.cc:348] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.853526 29376 raft_consensus.cc:2454] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:13.853832 30261 raft_consensus.cc:686] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:13.854183 30323 raft_consensus.cc:374] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.854301 30323 raft_consensus.cc:729] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.854159 30261 consensus_queue.cc:237] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:13.854557 30323 consensus_queue.cc:260] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.855255 30323 ts_tablet_manager.cc:1398] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:13.858606 29101 catalog_manager.cc:5484] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:13.922952 30309 raft_consensus.cc:482] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:13.923172 30309 raft_consensus.cc:504] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.924041 30309 leader_election.cc:288] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:13.924746 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:13.924727 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:13.925009 29506 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:13.925030 29376 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:13.925467 29181 leader_election.cc:302] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:13.925768 30309 raft_consensus.cc:2781] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:13.925880 30309 raft_consensus.cc:482] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:13.926019 30309 raft_consensus.cc:3037] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.928018 30309 raft_consensus.cc:504] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.928609 30309 leader_election.cc:288] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:13.929152 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:13.929229 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:13.929387 29376 raft_consensus.cc:3037] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.929423 29506 raft_consensus.cc:3037] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.931488 29506 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:13.931944 29180 leader_election.cc:302] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:13.932211 29376 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:13.932256 30309 raft_consensus.cc:2781] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:13.932499 30309 raft_consensus.cc:686] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:13.932802 30309 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:13.935715 29101 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:14.176435 30261 raft_consensus.cc:482] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.176612 30261 raft_consensus.cc:504] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:14.177245 30261 leader_election.cc:288] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:14.177764 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:14.177796 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:14.177953 29376 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:14.178107 29637 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:14.178324 29442 leader_election.cc:302] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:14.178592 30261 raft_consensus.cc:2781] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:14.178714 30261 raft_consensus.cc:482] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:14.178884 30261 raft_consensus.cc:3037] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.180745 30261 raft_consensus.cc:504] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:14.181339 30261 leader_election.cc:288] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:14.181881 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:14.181870 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:14.182091 29376 raft_consensus.cc:3037] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.182117 29637 raft_consensus.cc:3037] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.184856 29637 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:14.184857 29376 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:14.185349 29440 leader_election.cc:302] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:14.185608 30261 raft_consensus.cc:2781] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:14.185750 30261 raft_consensus.cc:686] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:14.186013 30261 consensus_queue.cc:237] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:14.188632 29101 catalog_manager.cc:5484] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:14.276157 30261 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.281569 30261 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.466068 30309 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.477635 30347 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:14.601949 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:14.602277 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:14.610517 30340 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:00:14.706090 30261 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.721158 30261 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.112 ms
time per row: 0.107556 ms
I20240501 14:00:14.985478 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:14.994302 29355 tablet_service.cc:1467] Processing CreateTablet for tablet f911e40875f2453e902e4cf1816578a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.994503 29226 tablet_service.cc:1467] Processing CreateTablet for tablet f911e40875f2453e902e4cf1816578a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.994832 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911e40875f2453e902e4cf1816578a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.994877 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911e40875f2453e902e4cf1816578a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.995257 29616 tablet_service.cc:1467] Processing CreateTablet for tablet f911e40875f2453e902e4cf1816578a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.995708 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911e40875f2453e902e4cf1816578a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.996007 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 770a6a8f192c430aa00e921af46543ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.996462 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770a6a8f192c430aa00e921af46543ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.996598 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 18da3f23b58c47468987acd6d5b04269 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.997077 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18da3f23b58c47468987acd6d5b04269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.001770 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 770a6a8f192c430aa00e921af46543ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.002321 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770a6a8f192c430aa00e921af46543ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.005744 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 18da3f23b58c47468987acd6d5b04269 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.006188 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18da3f23b58c47468987acd6d5b04269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.006381 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 18da3f23b58c47468987acd6d5b04269 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.006924 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18da3f23b58c47468987acd6d5b04269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.009433 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 770a6a8f192c430aa00e921af46543ba (DEFAULT_TABLE table=rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.009874 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770a6a8f192c430aa00e921af46543ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.009953 30405 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:15.014123 30405 tablet_bootstrap.cc:654] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.017802 30406 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:15.019471 30406 tablet_bootstrap.cc:654] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.022099 30406 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:15.022245 30406 ts_tablet_manager.cc:1367] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:15.022953 30406 raft_consensus.cc:348] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.023164 30406 raft_consensus.cc:374] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.023247 30406 raft_consensus.cc:729] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.023485 30406 consensus_queue.cc:260] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.024048 30406 ts_tablet_manager.cc:1398] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:15.032845 30409 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:15.034885 30409 tablet_bootstrap.cc:654] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.036613 30404 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:15.037050 30405 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:15.037225 30405 ts_tablet_manager.cc:1367] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20240501 14:00:15.038012 30405 raft_consensus.cc:348] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.038204 30405 raft_consensus.cc:374] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.038275 30405 raft_consensus.cc:729] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.038575 30404 tablet_bootstrap.cc:654] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.038484 30405 consensus_queue.cc:260] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.039027 30405 ts_tablet_manager.cc:1398] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:15.039290 30405 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:15.040805 30405 tablet_bootstrap.cc:654] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.045495 30405 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:15.045715 30405 ts_tablet_manager.cc:1367] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:00:15.046738 30405 raft_consensus.cc:348] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.047012 30405 raft_consensus.cc:374] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.047089 30405 raft_consensus.cc:729] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.047403 30405 consensus_queue.cc:260] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.048086 30405 ts_tablet_manager.cc:1398] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:15.048422 30405 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:15.050341 30405 tablet_bootstrap.cc:654] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.050658 30409 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:15.050799 30409 ts_tablet_manager.cc:1367] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240501 14:00:15.051810 30409 raft_consensus.cc:348] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.052098 30409 raft_consensus.cc:374] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.052197 30409 raft_consensus.cc:729] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.052462 30409 consensus_queue.cc:260] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.053398 30409 ts_tablet_manager.cc:1398] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:15.053712 30409 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:15.054332 30405 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:15.054484 30405 ts_tablet_manager.cc:1367] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:00:15.054980 30404 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:15.055193 30404 ts_tablet_manager.cc:1367] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20240501 14:00:15.055333 30409 tablet_bootstrap.cc:654] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.055214 30405 raft_consensus.cc:348] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.055429 30405 raft_consensus.cc:374] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.055532 30405 raft_consensus.cc:729] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.055781 30405 consensus_queue.cc:260] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.056473 30405 ts_tablet_manager.cc:1398] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:15.057188 30404 raft_consensus.cc:348] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.057464 30404 raft_consensus.cc:374] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.057999 30404 raft_consensus.cc:729] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.058236 30409 tablet_bootstrap.cc:492] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:15.058387 30409 ts_tablet_manager.cc:1367] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:00:15.058272 30404 consensus_queue.cc:260] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.059134 30409 raft_consensus.cc:348] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.059370 30409 raft_consensus.cc:374] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.059412 30404 ts_tablet_manager.cc:1398] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:15.059461 30409 raft_consensus.cc:729] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.059697 30404 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:15.059695 30409 consensus_queue.cc:260] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.060170 30409 ts_tablet_manager.cc:1398] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:15.060427 30409 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:15.061411 30404 tablet_bootstrap.cc:654] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.061717 30409 tablet_bootstrap.cc:654] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.066030 30409 tablet_bootstrap.cc:492] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:15.066160 30409 ts_tablet_manager.cc:1367] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 14:00:15.066249 30404 tablet_bootstrap.cc:492] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:15.066409 30404 ts_tablet_manager.cc:1367] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:00:15.066793 30409 raft_consensus.cc:348] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.066942 30409 raft_consensus.cc:374] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.067013 30409 raft_consensus.cc:729] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.066982 30404 raft_consensus.cc:348] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.067189 30404 raft_consensus.cc:374] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.067267 30404 raft_consensus.cc:729] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.067222 30409 consensus_queue.cc:260] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.067510 30404 consensus_queue.cc:260] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.067878 30409 ts_tablet_manager.cc:1398] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:15.068025 30404 ts_tablet_manager.cc:1398] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:15.145159 30245 raft_consensus.cc:482] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.145359 30245 raft_consensus.cc:504] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.146950 30245 leader_election.cc:288] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:15.155346 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911e40875f2453e902e4cf1816578a6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:15.155510 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911e40875f2453e902e4cf1816578a6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:15.155645 29246 raft_consensus.cc:2454] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:15.155754 29376 raft_consensus.cc:2454] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:15.156298 29570 leader_election.cc:302] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:15.156692 30245 raft_consensus.cc:2781] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.156822 30245 raft_consensus.cc:482] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.156924 30245 raft_consensus.cc:3037] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.158880 30245 raft_consensus.cc:504] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.159510 30245 leader_election.cc:288] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:15.159931 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911e40875f2453e902e4cf1816578a6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:15.160039 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911e40875f2453e902e4cf1816578a6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:15.160188 29376 raft_consensus.cc:3037] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.160266 29246 raft_consensus.cc:3037] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.163194 29376 raft_consensus.cc:2454] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:15.163208 29246 raft_consensus.cc:2454] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:15.163671 29572 leader_election.cc:302] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:15.163966 30245 raft_consensus.cc:2781] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.164460 30245 raft_consensus.cc:686] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:15.165028 30245 consensus_queue.cc:237] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.168937 29099 catalog_manager.cc:5484] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:15.185451 30396 raft_consensus.cc:482] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.185650 30396 raft_consensus.cc:504] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.186369 30396 leader_election.cc:288] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:15.186882 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770a6a8f192c430aa00e921af46543ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:15.186923 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770a6a8f192c430aa00e921af46543ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:15.187105 29636 raft_consensus.cc:2454] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:15.187131 29376 raft_consensus.cc:2454] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:15.187567 29181 leader_election.cc:302] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:15.187849 30396 raft_consensus.cc:2781] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.187951 30396 raft_consensus.cc:482] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.188015 30396 raft_consensus.cc:3037] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.190012 30396 raft_consensus.cc:504] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.190572 30396 leader_election.cc:288] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:15.191045 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770a6a8f192c430aa00e921af46543ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:15.191123 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770a6a8f192c430aa00e921af46543ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:15.191231 29636 raft_consensus.cc:3037] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.191318 29376 raft_consensus.cc:3037] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.193449 29636 raft_consensus.cc:2454] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:15.193619 29376 raft_consensus.cc:2454] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:15.193962 29179 leader_election.cc:302] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:15.194279 30396 raft_consensus.cc:2781] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.194451 30396 raft_consensus.cc:686] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:15.194723 30396 consensus_queue.cc:237] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:15.197564 29099 catalog_manager.cc:5484] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:15.225467 30390 raft_consensus.cc:482] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.225654 30390 raft_consensus.cc:504] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.226255 30245 raft_consensus.cc:482] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.226369 30390 leader_election.cc:288] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:15.226475 30245 raft_consensus.cc:504] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.226980 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:15.226984 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:15.227200 29636 raft_consensus.cc:2454] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:15.227238 29246 raft_consensus.cc:2454] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:15.227550 29440 leader_election.cc:302] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:15.227689 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:15.227913 29246 raft_consensus.cc:2454] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:15.228072 30390 raft_consensus.cc:2781] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.228205 30390 raft_consensus.cc:482] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.228310 30390 raft_consensus.cc:3037] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.228333 29570 leader_election.cc:302] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:15.228410 30245 leader_election.cc:288] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:15.228622 30245 raft_consensus.cc:2781] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.228703 30245 raft_consensus.cc:482] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.228804 30245 raft_consensus.cc:3037] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.230816 30390 raft_consensus.cc:504] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.231369 30390 leader_election.cc:288] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:15.231523 30245 raft_consensus.cc:504] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.231923 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:15.232161 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:15.232257 29636 raft_consensus.cc:2379] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 522f4ff5e9084cbf8462196533c7d7aa in current term 1: Already voted for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in this term.
I20240501 14:00:15.232378 29246 raft_consensus.cc:3037] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.232564 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:15.232771 30245 leader_election.cc:288] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:15.234989 29246 raft_consensus.cc:2454] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:15.235599 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:15.235694 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18da3f23b58c47468987acd6d5b04269" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:15.235780 29440 leader_election.cc:302] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:15.235917 29506 raft_consensus.cc:2379] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecc8bf0965164f199fb0bfa3d7b47c0e in current term 1: Already voted for candidate 522f4ff5e9084cbf8462196533c7d7aa in this term.
I20240501 14:00:15.236197 30390 raft_consensus.cc:2781] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.236290 29571 leader_election.cc:302] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:00:15.236342 30390 raft_consensus.cc:686] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:15.236546 30245 raft_consensus.cc:2726] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:00:15.236618 30390 consensus_queue.cc:237] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:15.239420 29099 catalog_manager.cc:5484] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:15.589998 30245 consensus_queue.cc:1035] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.595827 30245 consensus_queue.cc:1035] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:15.595705 30424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:15.596148 30424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:15.605687 30424 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:15.624107 30396 consensus_queue.cc:1035] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.628960 30396 consensus_queue.cc:1035] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 14:00:15.731127 30390 consensus_queue.cc:1035] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:15.815217 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 48516.8us
I20240501 14:00:15.846354 30389 consensus_queue.cc:1035] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.888118 30478 mvcc.cc:205] Tried to move back new op lower bound from 7022886976801783808 to 7022886974411657216. Current Snapshot: MvccSnapshot[applied={T|T < 7022886976576819200}]
INSERT report
rows total: 2000
time total: 287.278 ms
time per row: 0.143639 ms
I20240501 14:00:16.041930 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:16.050722 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.051167 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.051048 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.051497 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.052477 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 9d39eab213cc464ba90149276cfeeac8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.052892 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d39eab213cc464ba90149276cfeeac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.052855 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 0efddc01769f43e0b6f44d40ce6fa562 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.053303 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0efddc01769f43e0b6f44d40ce6fa562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.053454 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.053844 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.057945 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 9d39eab213cc464ba90149276cfeeac8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.058198 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 9d39eab213cc464ba90149276cfeeac8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.058320 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d39eab213cc464ba90149276cfeeac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.058574 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d39eab213cc464ba90149276cfeeac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.061667 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 0efddc01769f43e0b6f44d40ce6fa562 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.062054 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0efddc01769f43e0b6f44d40ce6fa562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.064517 29486 tablet_service.cc:1467] Processing CreateTablet for tablet 0efddc01769f43e0b6f44d40ce6fa562 (DEFAULT_TABLE table=rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:16.064939 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0efddc01769f43e0b6f44d40ce6fa562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.081291 30490 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:16.081450 30492 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:16.082527 30489 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:16.083611 30492 tablet_bootstrap.cc:654] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.083719 30490 tablet_bootstrap.cc:654] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.084371 30489 tablet_bootstrap.cc:654] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.086056 30491 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:16.087958 30490 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:16.088037 30492 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:16.088119 30490 ts_tablet_manager.cc:1367] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:00:16.088186 30492 ts_tablet_manager.cc:1367] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 14:00:16.088428 30491 tablet_bootstrap.cc:654] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.089105 30489 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:16.089104 30490 raft_consensus.cc:348] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.089332 30490 raft_consensus.cc:374] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.089175 30492 raft_consensus.cc:348] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.089432 30490 raft_consensus.cc:729] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.089430 30492 raft_consensus.cc:374] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.089555 30492 raft_consensus.cc:729] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.089501 30489 ts_tablet_manager.cc:1367] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:00:16.089727 30490 consensus_queue.cc:260] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.090539 30490 ts_tablet_manager.cc:1398] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:16.090693 30489 raft_consensus.cc:348] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.090849 30490 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:16.089905 30492 consensus_queue.cc:260] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.090979 30489 raft_consensus.cc:374] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.091167 30489 raft_consensus.cc:729] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.091495 30489 consensus_queue.cc:260] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.092732 30489 ts_tablet_manager.cc:1398] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:16.093112 30489 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:16.094077 30490 tablet_bootstrap.cc:654] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.097054 30489 tablet_bootstrap.cc:654] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.103077 30490 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:16.103474 30490 ts_tablet_manager.cc:1367] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240501 14:00:16.105123 30490 raft_consensus.cc:348] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.105361 30490 raft_consensus.cc:374] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.105441 30490 raft_consensus.cc:729] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.105688 30489 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:16.105912 30489 ts_tablet_manager.cc:1367] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20240501 14:00:16.105899 30490 consensus_queue.cc:260] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.106614 30490 ts_tablet_manager.cc:1398] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:16.106921 30489 raft_consensus.cc:348] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.107213 30489 raft_consensus.cc:374] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.107343 30489 raft_consensus.cc:729] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.107601 30489 consensus_queue.cc:260] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.108134 30489 ts_tablet_manager.cc:1398] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:16.108839 30492 ts_tablet_manager.cc:1398] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:00:16.109267 30492 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:16.111541 30492 tablet_bootstrap.cc:654] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.114327 30492 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:16.114511 30492 ts_tablet_manager.cc:1367] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 14:00:16.114534 30491 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:16.114682 30491 ts_tablet_manager.cc:1367] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.023s
I20240501 14:00:16.115367 30492 raft_consensus.cc:348] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.115440 30491 raft_consensus.cc:348] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.115613 30492 raft_consensus.cc:374] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.115638 30491 raft_consensus.cc:374] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.115711 30492 raft_consensus.cc:729] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.115736 30491 raft_consensus.cc:729] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.115953 30491 consensus_queue.cc:260] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.115970 30492 consensus_queue.cc:260] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.116550 30491 ts_tablet_manager.cc:1398] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:16.116552 30492 ts_tablet_manager.cc:1398] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:16.116842 30492 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:16.117225 30491 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:16.118548 30492 tablet_bootstrap.cc:654] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.119372 30491 tablet_bootstrap.cc:654] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:16.122112 30491 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:16.122148 30492 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:16.122259 30492 ts_tablet_manager.cc:1367] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 14:00:16.122279 30491 ts_tablet_manager.cc:1367] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:00:16.123015 30492 raft_consensus.cc:348] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.123003 30491 raft_consensus.cc:348] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.123222 30492 raft_consensus.cc:374] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.123239 30491 raft_consensus.cc:374] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:16.123319 30492 raft_consensus.cc:729] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.123307 30491 raft_consensus.cc:729] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:16.123556 30491 consensus_queue.cc:260] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.123536 30492 consensus_queue.cc:260] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.124150 30492 ts_tablet_manager.cc:1398] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:16.124250 30491 ts_tablet_manager.cc:1398] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:16.194447 30428 raft_consensus.cc:482] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:16.194649 30428 raft_consensus.cc:504] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.196213 30428 leader_election.cc:288] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:16.201891 30428 raft_consensus.cc:482] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:16.202086 30428 raft_consensus.cc:504] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.202939 30428 leader_election.cc:288] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:16.203068 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0efddc01769f43e0b6f44d40ce6fa562" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:16.203085 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0efddc01769f43e0b6f44d40ce6fa562" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:16.203318 29506 raft_consensus.cc:2454] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:16.203299 29636 raft_consensus.cc:2454] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:16.204044 29310 leader_election.cc:302] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:16.204191 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed6fab7d9e46aa8faeb3cf784af0fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:16.204394 29506 raft_consensus.cc:2454] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:16.204492 30428 raft_consensus.cc:2781] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:16.204578 30428 raft_consensus.cc:482] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:16.204694 30428 raft_consensus.cc:3037] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.206797 30428 raft_consensus.cc:504] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.207569 30428 leader_election.cc:288] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:16.207995 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0efddc01769f43e0b6f44d40ce6fa562" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:16.208178 29636 raft_consensus.cc:3037] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.208559 29311 leader_election.cc:302] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:16.208844 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0efddc01769f43e0b6f44d40ce6fa562" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:16.208931 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed6fab7d9e46aa8faeb3cf784af0fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:16.209141 29506 raft_consensus.cc:3037] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.209180 29246 raft_consensus.cc:2454] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:16.209362 30428 raft_consensus.cc:2781] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:16.209467 30428 raft_consensus.cc:482] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:16.209561 30428 raft_consensus.cc:3037] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.211786 29636 raft_consensus.cc:2454] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:16.211874 29506 raft_consensus.cc:2454] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:16.212199 29310 leader_election.cc:302] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:16.212154 30428 raft_consensus.cc:504] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.212918 30428 leader_election.cc:288] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:16.213008 30504 raft_consensus.cc:2781] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:16.213188 30504 raft_consensus.cc:686] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:16.213438 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed6fab7d9e46aa8faeb3cf784af0fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:16.213477 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed6fab7d9e46aa8faeb3cf784af0fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:16.213625 29246 raft_consensus.cc:3037] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.213673 29506 raft_consensus.cc:3037] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.213799 30504 consensus_queue.cc:237] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.215670 29246 raft_consensus.cc:2454] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:16.216192 29310 leader_election.cc:302] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:16.216524 30504 raft_consensus.cc:2781] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:16.216529 29506 raft_consensus.cc:2454] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:16.216868 30504 raft_consensus.cc:686] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:16.217172 30504 consensus_queue.cc:237] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:16.217689 29101 catalog_manager.cc:5484] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:16.222481 29101 catalog_manager.cc:5484] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:16.247015 30347 raft_consensus.cc:482] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:16.247236 30347 raft_consensus.cc:504] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.247946 30347 leader_election.cc:288] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:16.248579 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d39eab213cc464ba90149276cfeeac8" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:16.248592 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d39eab213cc464ba90149276cfeeac8" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:16.248814 29506 raft_consensus.cc:2454] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:16.248812 29636 raft_consensus.cc:2454] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:16.249258 29179 leader_election.cc:302] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:16.249541 30347 raft_consensus.cc:2781] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:16.249624 30347 raft_consensus.cc:482] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:16.249699 30347 raft_consensus.cc:3037] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.251477 30347 raft_consensus.cc:504] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.252069 30347 leader_election.cc:288] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:16.252542 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d39eab213cc464ba90149276cfeeac8" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:16.252557 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d39eab213cc464ba90149276cfeeac8" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:16.252740 29506 raft_consensus.cc:3037] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.252750 29636 raft_consensus.cc:3037] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:16.254679 29636 raft_consensus.cc:2454] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:16.254684 29506 raft_consensus.cc:2454] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:16.255111 29180 leader_election.cc:302] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:16.255445 30347 raft_consensus.cc:2781] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:16.255615 30347 raft_consensus.cc:686] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:16.255923 30347 consensus_queue.cc:237] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:16.258894 29101 catalog_manager.cc:5484] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:16.630275 30504 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.641122 30504 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:16.654158 30508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:16.654484 30508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:16.662818 30508 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:16.672223 30428 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.673249 30428 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 14:00:16.776427 30347 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.785926 30347 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.549 ms
time per row: 0.121274 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:18.021437 29102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:18.030876 29224 tablet_service.cc:1467] Processing CreateTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.030870 29487 tablet_service.cc:1467] Processing CreateTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.031307 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09ea61ef29d4e1bb1a20d866b1e8051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.031371 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09ea61ef29d4e1bb1a20d866b1e8051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.032061 29357 tablet_service.cc:1467] Processing CreateTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.032474 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09ea61ef29d4e1bb1a20d866b1e8051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.033604 29355 tablet_service.cc:1467] Processing CreateTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.033985 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cebf8b40474bf2b500e15fbf3c669c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.036747 29225 tablet_service.cc:1467] Processing CreateTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.036887 30574 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:18.037153 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cebf8b40474bf2b500e15fbf3c669c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.038674 30574 tablet_bootstrap.cc:654] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.041574 29616 tablet_service.cc:1467] Processing CreateTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.042096 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cebf8b40474bf2b500e15fbf3c669c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.046340 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 5be28b16ee2a4594b53a99ca80367596 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.046761 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be28b16ee2a4594b53a99ca80367596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.045609 30577 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:18.046916 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 5be28b16ee2a4594b53a99ca80367596 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.047298 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be28b16ee2a4594b53a99ca80367596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.048593 30577 tablet_bootstrap.cc:654] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.050277 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 5be28b16ee2a4594b53a99ca80367596 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.050737 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be28b16ee2a4594b53a99ca80367596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.060464 30575 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:18.060707 30577 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:18.061168 30577 ts_tablet_manager.cc:1367] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240501 14:00:18.062108 30577 raft_consensus.cc:348] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.062327 30575 tablet_bootstrap.cc:654] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.062376 30577 raft_consensus.cc:374] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.062484 30577 raft_consensus.cc:729] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.062789 30577 consensus_queue.cc:260] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.063535 30577 ts_tablet_manager.cc:1398] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:18.063925 30577 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:18.065992 30577 tablet_bootstrap.cc:654] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.068432 30580 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:18.070786 30580 tablet_bootstrap.cc:654] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.074757 30577 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:18.075048 30577 ts_tablet_manager.cc:1367] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:00:18.076478 30575 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:18.076666 30574 tablet_bootstrap.cc:492] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:18.076689 30575 ts_tablet_manager.cc:1367] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240501 14:00:18.077010 30574 ts_tablet_manager.cc:1367] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.014s
I20240501 14:00:18.076959 30577 raft_consensus.cc:348] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.077322 30577 raft_consensus.cc:374] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.077455 30577 raft_consensus.cc:729] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.077781 30577 consensus_queue.cc:260] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.078105 30574 raft_consensus.cc:348] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.078327 30574 raft_consensus.cc:374] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.078435 30574 raft_consensus.cc:729] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.078702 30577 ts_tablet_manager.cc:1398] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:18.078662 30574 consensus_queue.cc:260] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.078784 30575 raft_consensus.cc:348] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.079007 30575 raft_consensus.cc:374] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.079130 30575 raft_consensus.cc:729] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.079391 30575 consensus_queue.cc:260] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.079896 30574 ts_tablet_manager.cc:1398] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:18.080020 30575 ts_tablet_manager.cc:1398] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:18.080345 30574 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:18.080348 30575 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:18.082067 30574 tablet_bootstrap.cc:654] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.082469 30575 tablet_bootstrap.cc:654] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.087381 30580 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:18.087656 30580 ts_tablet_manager.cc:1367] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.003s
I20240501 14:00:18.088744 30580 raft_consensus.cc:348] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.089061 30580 raft_consensus.cc:374] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.089190 30580 raft_consensus.cc:729] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.089461 30580 consensus_queue.cc:260] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.090227 30580 ts_tablet_manager.cc:1398] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:18.090690 30580 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:18.092805 30580 tablet_bootstrap.cc:654] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.093958 30575 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:18.094229 30575 ts_tablet_manager.cc:1367] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240501 14:00:18.095444 30575 raft_consensus.cc:348] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.095695 30575 raft_consensus.cc:374] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.095768 30575 raft_consensus.cc:729] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.096040 30575 consensus_queue.cc:260] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.096647 30574 tablet_bootstrap.cc:492] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:18.096662 30575 ts_tablet_manager.cc:1398] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:18.096951 30574 ts_tablet_manager.cc:1367] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20240501 14:00:18.097090 30580 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:18.097272 30580 ts_tablet_manager.cc:1367] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:00:18.097887 30574 raft_consensus.cc:348] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.098170 30574 raft_consensus.cc:374] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.098059 30580 raft_consensus.cc:348] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.098266 30574 raft_consensus.cc:729] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.098317 30580 raft_consensus.cc:374] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.098395 30580 raft_consensus.cc:729] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.098574 30574 consensus_queue.cc:260] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.098646 30580 consensus_queue.cc:260] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.099239 30580 ts_tablet_manager.cc:1398] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:18.099880 30574 ts_tablet_manager.cc:1398] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:18.100201 30574 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:18.102020 30574 tablet_bootstrap.cc:654] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.105115 30574 tablet_bootstrap.cc:492] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:18.105270 30574 ts_tablet_manager.cc:1367] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:00:18.106001 30574 raft_consensus.cc:348] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.106169 30574 raft_consensus.cc:374] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.106254 30574 raft_consensus.cc:729] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.106460 30574 consensus_queue.cc:260] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.106946 30574 ts_tablet_manager.cc:1398] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:18.251803 30569 raft_consensus.cc:482] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.251987 30569 raft_consensus.cc:504] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.252624 30569 leader_election.cc:288] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:18.253254 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09ea61ef29d4e1bb1a20d866b1e8051" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:18.253265 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09ea61ef29d4e1bb1a20d866b1e8051" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:18.253479 29506 raft_consensus.cc:2454] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:18.253535 29246 raft_consensus.cc:2454] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:18.254004 29310 leader_election.cc:302] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:18.254354 30569 raft_consensus.cc:2781] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.254451 30569 raft_consensus.cc:482] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.254539 30569 raft_consensus.cc:3037] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.256916 30569 raft_consensus.cc:504] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.257503 30569 leader_election.cc:288] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:18.257920 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09ea61ef29d4e1bb1a20d866b1e8051" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:18.257953 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09ea61ef29d4e1bb1a20d866b1e8051" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:18.258088 29506 raft_consensus.cc:3037] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.258144 29246 raft_consensus.cc:3037] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.259915 29506 raft_consensus.cc:2454] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:18.259959 29246 raft_consensus.cc:2454] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:18.260293 29311 leader_election.cc:302] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:18.260612 30569 raft_consensus.cc:2781] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.260756 30569 raft_consensus.cc:686] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:18.261082 30569 consensus_queue.cc:237] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.263810 29098 catalog_manager.cc:5484] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:18.300810 30389 raft_consensus.cc:482] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.301009 30389 raft_consensus.cc:504] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.301743 30389 leader_election.cc:288] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:18.302256 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be28b16ee2a4594b53a99ca80367596" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:18.302340 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be28b16ee2a4594b53a99ca80367596" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:18.302477 29637 raft_consensus.cc:2454] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:18.302608 29246 raft_consensus.cc:2454] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:18.302959 29440 leader_election.cc:302] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:18.303216 30389 raft_consensus.cc:2781] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.303331 30389 raft_consensus.cc:482] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.303416 30389 raft_consensus.cc:3037] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.305572 30389 raft_consensus.cc:504] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.306084 30389 leader_election.cc:288] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:18.306484 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be28b16ee2a4594b53a99ca80367596" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:18.306563 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be28b16ee2a4594b53a99ca80367596" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:18.306661 29637 raft_consensus.cc:3037] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.306757 29246 raft_consensus.cc:3037] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.308562 29637 raft_consensus.cc:2454] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:18.308696 29246 raft_consensus.cc:2454] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:18.308954 29440 leader_election.cc:302] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:18.309324 30389 raft_consensus.cc:2781] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.309670 30389 raft_consensus.cc:686] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:18.309937 30389 consensus_queue.cc:237] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:18.312923 29098 catalog_manager.cc:5484] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:18.353102 30541 raft_consensus.cc:482] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.353292 30541 raft_consensus.cc:504] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.354003 30541 leader_election.cc:288] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:18.354480 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cebf8b40474bf2b500e15fbf3c669c" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:18.354568 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cebf8b40474bf2b500e15fbf3c669c" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:18.354701 29637 raft_consensus.cc:2454] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:18.354792 29375 raft_consensus.cc:2454] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:18.355118 29179 leader_election.cc:302] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:18.355386 30541 raft_consensus.cc:2781] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.355496 30541 raft_consensus.cc:482] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.355573 30541 raft_consensus.cc:3037] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.357471 30541 raft_consensus.cc:504] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.358160 30541 leader_election.cc:288] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:18.358515 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cebf8b40474bf2b500e15fbf3c669c" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:18.358690 29637 raft_consensus.cc:3037] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.358664 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cebf8b40474bf2b500e15fbf3c669c" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:18.358870 29375 raft_consensus.cc:3037] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.360694 29637 raft_consensus.cc:2454] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:18.360883 29375 raft_consensus.cc:2454] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:18.361128 29179 leader_election.cc:302] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:18.361413 30541 raft_consensus.cc:2781] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.361811 30541 raft_consensus.cc:686] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:18.362181 30541 consensus_queue.cc:237] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:18.364861 29098 catalog_manager.cc:5484] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:18.710356 30569 consensus_queue.cc:1035] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.716202 30587 consensus_queue.cc:1035] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:18.772825 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:18.773196 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:18.780736 30593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:18.802531 30389 consensus_queue.cc:1035] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.807258 30589 consensus_queue.cc:1035] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 14:00:18.872087 30541 consensus_queue.cc:1035] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.891384 30624 consensus_queue.cc:1035] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 275.63 ms
time per row: 0.137815 ms
I20240501 14:00:19.219957 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:19.228863 29487 tablet_service.cc:1467] Processing CreateTablet for tablet a87b1de51c4c4f909c27c27aa745fb44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.228858 29357 tablet_service.cc:1467] Processing CreateTablet for tablet a87b1de51c4c4f909c27c27aa745fb44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.229390 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87b1de51c4c4f909c27c27aa745fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.229368 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87b1de51c4c4f909c27c27aa745fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.229571 29485 tablet_service.cc:1467] Processing CreateTablet for tablet d2e5851626b54e1caf6105a70ccafc43 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.229970 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e5851626b54e1caf6105a70ccafc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.230648 29617 tablet_service.cc:1467] Processing CreateTablet for tablet a87b1de51c4c4f909c27c27aa745fb44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.231047 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87b1de51c4c4f909c27c27aa745fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.237421 29616 tablet_service.cc:1467] Processing CreateTablet for tablet d2e5851626b54e1caf6105a70ccafc43 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.237699 29224 tablet_service.cc:1467] Processing CreateTablet for tablet c6973301bdd94c2b8f98e718715ff752 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.237902 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e5851626b54e1caf6105a70ccafc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.238085 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6973301bdd94c2b8f98e718715ff752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.239776 30660 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:19.242033 29486 tablet_service.cc:1467] Processing CreateTablet for tablet c6973301bdd94c2b8f98e718715ff752 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.242385 29355 tablet_service.cc:1467] Processing CreateTablet for tablet d2e5851626b54e1caf6105a70ccafc43 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.242545 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6973301bdd94c2b8f98e718715ff752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.242772 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e5851626b54e1caf6105a70ccafc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.245460 29356 tablet_service.cc:1467] Processing CreateTablet for tablet c6973301bdd94c2b8f98e718715ff752 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.245954 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6973301bdd94c2b8f98e718715ff752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.250617 30661 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:19.252424 30661 tablet_bootstrap.cc:654] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.252861 30660 tablet_bootstrap.cc:654] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.254698 30659 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:19.255772 30660 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:19.255980 30660 ts_tablet_manager.cc:1367] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 14:00:19.256806 30659 tablet_bootstrap.cc:654] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.256870 30660 raft_consensus.cc:348] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.257130 30660 raft_consensus.cc:374] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.257261 30660 raft_consensus.cc:729] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.257557 30660 consensus_queue.cc:260] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.258242 30660 ts_tablet_manager.cc:1398] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:19.259028 30660 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:19.260649 30660 tablet_bootstrap.cc:654] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.270866 30660 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:19.272423 30660 ts_tablet_manager.cc:1367] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240501 14:00:19.272634 30666 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:19.273583 30661 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:19.273754 30661 ts_tablet_manager.cc:1367] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.017s
I20240501 14:00:19.273629 30660 raft_consensus.cc:348] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.273851 30660 raft_consensus.cc:374] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.273924 30660 raft_consensus.cc:729] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.274166 30660 consensus_queue.cc:260] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.274533 30666 tablet_bootstrap.cc:654] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.274714 30660 ts_tablet_manager.cc:1398] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:19.274995 30660 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:19.274566 30661 raft_consensus.cc:348] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.275568 30661 raft_consensus.cc:374] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.275736 30661 raft_consensus.cc:729] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.276428 30661 consensus_queue.cc:260] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.276623 30660 tablet_bootstrap.cc:654] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.278002 30661 ts_tablet_manager.cc:1398] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:19.279500 30661 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:19.283830 30661 tablet_bootstrap.cc:654] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.284404 30659 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:19.284688 30659 ts_tablet_manager.cc:1367] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.004s
I20240501 14:00:19.285907 30659 raft_consensus.cc:348] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.286689 30659 raft_consensus.cc:374] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.286863 30659 raft_consensus.cc:729] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.287238 30659 consensus_queue.cc:260] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.288136 30659 ts_tablet_manager.cc:1398] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:00:19.288452 30659 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:19.291354 30659 tablet_bootstrap.cc:654] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.294083 30659 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:19.294318 30659 ts_tablet_manager.cc:1367] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:00:19.295418 30659 raft_consensus.cc:348] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.295782 30659 raft_consensus.cc:374] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.295914 30659 raft_consensus.cc:729] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.296319 30659 consensus_queue.cc:260] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.297034 30659 ts_tablet_manager.cc:1398] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:19.297379 30659 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:19.300593 30659 tablet_bootstrap.cc:654] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.302245 30660 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:19.302477 30660 ts_tablet_manager.cc:1367] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s
I20240501 14:00:19.302697 30661 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:19.302904 30666 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:19.303020 30661 ts_tablet_manager.cc:1367] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20240501 14:00:19.303169 30666 ts_tablet_manager.cc:1367] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.005s
I20240501 14:00:19.303444 30660 raft_consensus.cc:348] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.303653 30660 raft_consensus.cc:374] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.303746 30660 raft_consensus.cc:729] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.303972 30660 consensus_queue.cc:260] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.304380 30569 raft_consensus.cc:482] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.304510 30569 raft_consensus.cc:504] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.304644 30660 ts_tablet_manager.cc:1398] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:19.305197 30569 leader_election.cc:288] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:19.304081 30661 raft_consensus.cc:348] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.305363 30661 raft_consensus.cc:374] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.305491 30661 raft_consensus.cc:729] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.305788 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87b1de51c4c4f909c27c27aa745fb44" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:19.305903 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87b1de51c4c4f909c27c27aa745fb44" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:19.306039 29637 raft_consensus.cc:2454] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:19.306102 29507 raft_consensus.cc:2454] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:19.306561 29310 leader_election.cc:302] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:19.306707 30661 consensus_queue.cc:260] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.306887 30569 raft_consensus.cc:2781] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.307014 30569 raft_consensus.cc:482] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.307101 30569 raft_consensus.cc:3037] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.309525 30661 ts_tablet_manager.cc:1398] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240501 14:00:19.310864 30569 raft_consensus.cc:504] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.311698 30569 leader_election.cc:288] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:19.312180 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87b1de51c4c4f909c27c27aa745fb44" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:19.312209 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87b1de51c4c4f909c27c27aa745fb44" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:19.312382 29507 raft_consensus.cc:3037] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.312417 29637 raft_consensus.cc:3037] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.315217 29507 raft_consensus.cc:2454] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:19.315202 29637 raft_consensus.cc:2454] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:19.315809 29311 leader_election.cc:302] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:19.316396 30654 raft_consensus.cc:2781] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.316570 30654 raft_consensus.cc:686] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:19.316906 30654 consensus_queue.cc:237] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:19.304265 30666 raft_consensus.cc:348] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.318301 30666 raft_consensus.cc:374] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.318485 30666 raft_consensus.cc:729] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.318884 30666 consensus_queue.cc:260] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.319921 30666 ts_tablet_manager.cc:1398] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240501 14:00:19.322535 29096 catalog_manager.cc:5484] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:19.335049 30389 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.335228 30389 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.335901 30389 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:19.336323 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:19.336388 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:19.336535 29637 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:19.336620 29375 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:19.337076 29442 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:19.337366 30389 raft_consensus.cc:2781] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.337503 30389 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.337523 30659 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:19.337594 30389 raft_consensus.cc:3037] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.337664 30659 ts_tablet_manager.cc:1367] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.014s
I20240501 14:00:19.338321 30659 raft_consensus.cc:348] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.338485 30659 raft_consensus.cc:374] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.338579 30659 raft_consensus.cc:729] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.338798 30659 consensus_queue.cc:260] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.339303 30659 ts_tablet_manager.cc:1398] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:19.339764 30389 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.340801 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:19.341035 29637 raft_consensus.cc:3037] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.341688 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:19.341910 29375 raft_consensus.cc:3037] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.343585 29637 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:19.343823 30389 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:19.344071 29440 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:19.344343 29375 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:19.344368 30389 raft_consensus.cc:2781] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.344507 30389 raft_consensus.cc:686] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:19.345067 30389 consensus_queue.cc:237] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.347834 29096 catalog_manager.cc:5484] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:19.372867 30650 raft_consensus.cc:482] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.373093 30650 raft_consensus.cc:504] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.373826 30650 leader_election.cc:288] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:19.374418 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6973301bdd94c2b8f98e718715ff752" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:19.374425 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6973301bdd94c2b8f98e718715ff752" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:19.374656 29375 raft_consensus.cc:2454] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:19.374670 29507 raft_consensus.cc:2454] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:19.375092 29180 leader_election.cc:302] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:19.375353 30650 raft_consensus.cc:2781] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.375499 30650 raft_consensus.cc:482] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.375587 30650 raft_consensus.cc:3037] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.377491 30650 raft_consensus.cc:504] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.378096 30650 leader_election.cc:288] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:19.378453 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6973301bdd94c2b8f98e718715ff752" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:19.378595 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6973301bdd94c2b8f98e718715ff752" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:19.378683 29507 raft_consensus.cc:3037] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.378824 29376 raft_consensus.cc:3037] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.380738 29376 raft_consensus.cc:2454] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:19.381124 29181 leader_election.cc:302] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:19.381402 30650 raft_consensus.cc:2781] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.381538 30650 raft_consensus.cc:686] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:19.381819 30650 consensus_queue.cc:237] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:19.384639 29096 catalog_manager.cc:5484] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:19.420197 29507 raft_consensus.cc:2454] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:19.730404 30389 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.737303 30589 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.756666 30654 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.761198 30569 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:19.772608 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:19.772927 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:19.780719 30675 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:00:19.962513 30650 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.004736 29507 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:20.005669 30650 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 249.784 ms
time per row: 0.124892 ms
I20240501 14:00:20.176950 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:20.186440 29356 tablet_service.cc:1467] Processing CreateTablet for tablet da05f71e097a4661b687e94d82b2133a (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.186908 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05f71e097a4661b687e94d82b2133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.186831 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.187291 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9173d6ec3fe44271bcec61a0b32d5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.187541 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 58828a3d248c4e57908eeaaf9f880adc (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.187894 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58828a3d248c4e57908eeaaf9f880adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.188288 29224 tablet_service.cc:1467] Processing CreateTablet for tablet da05f71e097a4661b687e94d82b2133a (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.188685 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05f71e097a4661b687e94d82b2133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.193393 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.193408 29617 tablet_service.cc:1467] Processing CreateTablet for tablet da05f71e097a4661b687e94d82b2133a (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.193861 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9173d6ec3fe44271bcec61a0b32d5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.193914 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05f71e097a4661b687e94d82b2133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.196833 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 58828a3d248c4e57908eeaaf9f880adc (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.197440 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58828a3d248c4e57908eeaaf9f880adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.197432 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.197908 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9173d6ec3fe44271bcec61a0b32d5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.201918 30736 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:20.202018 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 58828a3d248c4e57908eeaaf9f880adc (DEFAULT_TABLE table=rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.202452 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58828a3d248c4e57908eeaaf9f880adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.203763 30736 tablet_bootstrap.cc:654] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.210180 30736 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:20.210359 30736 ts_tablet_manager.cc:1367] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:00:20.211185 30736 raft_consensus.cc:348] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.211417 30736 raft_consensus.cc:374] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.211513 30736 raft_consensus.cc:729] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.211735 30736 consensus_queue.cc:260] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.212355 30736 ts_tablet_manager.cc:1398] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:20.213994 30738 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:20.215509 30738 tablet_bootstrap.cc:654] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.217029 30735 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:20.218819 30735 tablet_bootstrap.cc:654] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.219888 30740 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:20.221678 30740 tablet_bootstrap.cc:654] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.225123 30735 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:20.225216 30738 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:20.225256 30740 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:20.225272 30735 ts_tablet_manager.cc:1367] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240501 14:00:20.225376 30740 ts_tablet_manager.cc:1367] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:00:20.225376 30738 ts_tablet_manager.cc:1367] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240501 14:00:20.225920 30735 raft_consensus.cc:348] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.226095 30735 raft_consensus.cc:374] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.226182 30735 raft_consensus.cc:729] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.226104 30738 raft_consensus.cc:348] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.226100 30740 raft_consensus.cc:348] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.226289 30738 raft_consensus.cc:374] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.226290 30740 raft_consensus.cc:374] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.226388 30740 raft_consensus.cc:729] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.226388 30738 raft_consensus.cc:729] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.226487 30735 consensus_queue.cc:260] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.226603 30740 consensus_queue.cc:260] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.226603 30738 consensus_queue.cc:260] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.227234 30740 ts_tablet_manager.cc:1398] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:20.227309 30735 ts_tablet_manager.cc:1398] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:20.227489 30740 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:20.227617 30735 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:20.228578 30738 ts_tablet_manager.cc:1398] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:20.228843 30738 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:20.229346 30740 tablet_bootstrap.cc:654] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.230508 30738 tablet_bootstrap.cc:654] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.231956 30735 tablet_bootstrap.cc:654] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.238574 30735 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:20.238744 30735 ts_tablet_manager.cc:1367] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 14:00:20.238909 30738 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:20.239064 30738 ts_tablet_manager.cc:1367] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240501 14:00:20.239390 30735 raft_consensus.cc:348] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.239586 30735 raft_consensus.cc:374] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.239665 30735 raft_consensus.cc:729] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.239646 30738 raft_consensus.cc:348] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.239864 30738 raft_consensus.cc:374] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.239941 30738 raft_consensus.cc:729] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.239907 30735 consensus_queue.cc:260] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.240168 30738 consensus_queue.cc:260] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.240377 30735 ts_tablet_manager.cc:1398] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:20.240727 30738 ts_tablet_manager.cc:1398] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:20.241168 30738 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:20.241870 30740 tablet_bootstrap.cc:492] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:20.242101 30740 ts_tablet_manager.cc:1367] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240501 14:00:20.242767 30738 tablet_bootstrap.cc:654] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.243134 30740 raft_consensus.cc:348] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.243455 30740 raft_consensus.cc:374] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.243549 30740 raft_consensus.cc:729] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.245023 30740 consensus_queue.cc:260] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.245790 30740 ts_tablet_manager.cc:1398] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:20.246325 30740 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:20.248767 30740 tablet_bootstrap.cc:654] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.250627 30738 tablet_bootstrap.cc:492] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:20.250780 30738 ts_tablet_manager.cc:1367] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:00:20.251297 30740 tablet_bootstrap.cc:492] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:20.251416 30740 ts_tablet_manager.cc:1367] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:00:20.251390 30738 raft_consensus.cc:348] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.251593 30738 raft_consensus.cc:374] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.251688 30738 raft_consensus.cc:729] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.251942 30738 consensus_queue.cc:260] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.251997 30740 raft_consensus.cc:348] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.252147 30740 raft_consensus.cc:374] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.252401 30740 raft_consensus.cc:729] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.252504 30738 ts_tablet_manager.cc:1398] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:20.252611 30740 consensus_queue.cc:260] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.253582 30740 ts_tablet_manager.cc:1398] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:20.272650 30719 raft_consensus.cc:482] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:20.272843 30719 raft_consensus.cc:504] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.273556 30719 leader_election.cc:288] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:20.274142 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9173d6ec3fe44271bcec61a0b32d5c4d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:20.274174 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9173d6ec3fe44271bcec61a0b32d5c4d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:20.274428 29637 raft_consensus.cc:2454] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:20.274446 29506 raft_consensus.cc:2454] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:20.274850 29180 leader_election.cc:302] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:20.275156 30719 raft_consensus.cc:2781] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:20.275280 30719 raft_consensus.cc:482] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:20.275382 30719 raft_consensus.cc:3037] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.277622 30719 raft_consensus.cc:504] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.278254 30719 leader_election.cc:288] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:20.278771 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9173d6ec3fe44271bcec61a0b32d5c4d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:20.278774 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9173d6ec3fe44271bcec61a0b32d5c4d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:20.279001 29637 raft_consensus.cc:3037] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.279002 29506 raft_consensus.cc:3037] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.281172 30569 raft_consensus.cc:482] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:20.281351 30569 raft_consensus.cc:504] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.281914 29506 raft_consensus.cc:2454] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:20.281913 29637 raft_consensus.cc:2454] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:20.281978 30569 leader_election.cc:288] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:20.282377 29180 leader_election.cc:302] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:20.282533 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58828a3d248c4e57908eeaaf9f880adc" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:20.282725 30719 raft_consensus.cc:2781] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.282786 29244 raft_consensus.cc:2454] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:20.282757 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58828a3d248c4e57908eeaaf9f880adc" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:20.282879 30719 raft_consensus.cc:686] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:20.282971 29636 raft_consensus.cc:2454] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:20.283239 29310 leader_election.cc:302] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:20.283190 30719 consensus_queue.cc:237] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:20.283601 30569 raft_consensus.cc:2781] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:20.283687 30569 raft_consensus.cc:482] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:20.283814 30569 raft_consensus.cc:3037] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.286131 30569 raft_consensus.cc:504] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.286520 29102 catalog_manager.cc:5484] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:20.286800 30569 leader_election.cc:288] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:20.287249 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58828a3d248c4e57908eeaaf9f880adc" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:20.287290 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58828a3d248c4e57908eeaaf9f880adc" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:20.287449 29244 raft_consensus.cc:3037] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.287472 29636 raft_consensus.cc:3037] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.289868 29244 raft_consensus.cc:2454] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:20.289883 29636 raft_consensus.cc:2454] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:20.290273 29310 leader_election.cc:302] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:20.290546 30569 raft_consensus.cc:2781] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.290711 30569 raft_consensus.cc:686] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:20.290966 30569 consensus_queue.cc:237] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.293596 29099 catalog_manager.cc:5484] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:20.333981 30569 raft_consensus.cc:482] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:20.334151 30569 raft_consensus.cc:504] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.334713 30569 leader_election.cc:288] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:20.335186 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05f71e097a4661b687e94d82b2133a" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:20.335310 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05f71e097a4661b687e94d82b2133a" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:20.335434 29244 raft_consensus.cc:2454] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:20.335556 29636 raft_consensus.cc:2454] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:20.335873 29310 leader_election.cc:302] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:20.336170 30569 raft_consensus.cc:2781] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:20.336333 30569 raft_consensus.cc:482] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:20.336416 30569 raft_consensus.cc:3037] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.338523 30569 raft_consensus.cc:504] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.339078 30569 leader_election.cc:288] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:20.339491 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05f71e097a4661b687e94d82b2133a" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:20.339576 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05f71e097a4661b687e94d82b2133a" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:20.339685 29244 raft_consensus.cc:3037] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.339830 29636 raft_consensus.cc:3037] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.341790 29244 raft_consensus.cc:2454] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:20.341895 29636 raft_consensus.cc:2454] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:20.342239 29310 leader_election.cc:302] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:20.342530 30569 raft_consensus.cc:2781] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.342684 30569 raft_consensus.cc:686] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:20.342958 30569 consensus_queue.cc:237] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:20.345882 29099 catalog_manager.cc:5484] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:20.697510 30719 consensus_queue.cc:1035] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.705716 30755 consensus_queue.cc:1035] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.725970 30569 consensus_queue.cc:1035] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.731832 30569 consensus_queue.cc:1035] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:20.734159 30751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:20.734591 30751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:20.742947 30751 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:20.771406 30569 consensus_queue.cc:1035] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 14:00:20.776659 30569 consensus_queue.cc:1035] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 304.033 ms
time per row: 0.152016 ms
I20240501 14:00:21.199857 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:21.208561 29226 tablet_service.cc:1467] Processing CreateTablet for tablet ca0a888dc37d44c69dea73f9ffd556f6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.208740 29485 tablet_service.cc:1467] Processing CreateTablet for tablet ca0a888dc37d44c69dea73f9ffd556f6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.209074 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0a888dc37d44c69dea73f9ffd556f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.209308 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.209555 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0a888dc37d44c69dea73f9ffd556f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.209457 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.209889 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.210067 29355 tablet_service.cc:1467] Processing CreateTablet for tablet ca0a888dc37d44c69dea73f9ffd556f6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.210448 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0a888dc37d44c69dea73f9ffd556f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.214663 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.224635 30817 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:21.226819 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.227160 30817 tablet_bootstrap.cc:654] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.227274 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.227705 29617 tablet_service.cc:1467] Processing CreateTablet for tablet f345c7a43d2648b4b4c47e726c4d4c51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.228235 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345c7a43d2648b4b4c47e726c4d4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.230842 29485 tablet_service.cc:1467] Processing CreateTablet for tablet f345c7a43d2648b4b4c47e726c4d4c51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.231266 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345c7a43d2648b4b4c47e726c4d4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.237239 29356 tablet_service.cc:1467] Processing CreateTablet for tablet f345c7a43d2648b4b4c47e726c4d4c51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.237708 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345c7a43d2648b4b4c47e726c4d4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.248142 30816 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:21.249269 30820 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:21.249563 30817 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:21.249729 30817 ts_tablet_manager.cc:1367] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.010s
I20240501 14:00:21.250491 30817 raft_consensus.cc:348] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.250676 30817 raft_consensus.cc:374] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.250767 30817 raft_consensus.cc:729] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.251049 30817 consensus_queue.cc:260] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.251220 30816 tablet_bootstrap.cc:654] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.251225 30820 tablet_bootstrap.cc:654] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.251766 30817 ts_tablet_manager.cc:1398] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.252028 30817 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:21.252732 30818 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:21.254244 30817 tablet_bootstrap.cc:654] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.255571 30818 tablet_bootstrap.cc:654] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.258919 30820 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:21.259112 30820 ts_tablet_manager.cc:1367] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240501 14:00:21.260227 30820 raft_consensus.cc:348] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.260524 30820 raft_consensus.cc:374] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.260663 30820 raft_consensus.cc:729] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.261042 30820 consensus_queue.cc:260] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.263108 30816 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:21.263341 30820 ts_tablet_manager.cc:1398] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:21.263584 30817 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:21.263654 30820 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:21.263700 30816 ts_tablet_manager.cc:1367] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240501 14:00:21.263760 30817 ts_tablet_manager.cc:1367] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 14:00:21.264546 30816 raft_consensus.cc:348] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.264564 30817 raft_consensus.cc:348] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.264775 30816 raft_consensus.cc:374] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.264856 30816 raft_consensus.cc:729] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.264859 30817 raft_consensus.cc:374] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.264943 30817 raft_consensus.cc:729] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.265107 30816 consensus_queue.cc:260] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.265394 30820 tablet_bootstrap.cc:654] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.265873 30818 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:21.266039 30818 ts_tablet_manager.cc:1367] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240501 14:00:21.266160 30816 ts_tablet_manager.cc:1398] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:21.266479 30816 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:21.267045 30818 raft_consensus.cc:348] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.267272 30818 raft_consensus.cc:374] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.267362 30818 raft_consensus.cc:729] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.268143 30818 consensus_queue.cc:260] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.269502 30816 tablet_bootstrap.cc:654] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.271538 30818 ts_tablet_manager.cc:1398] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:00:21.272457 30818 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:21.274407 30818 tablet_bootstrap.cc:654] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.276512 30818 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:21.276674 30818 ts_tablet_manager.cc:1367] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:21.275851 30817 consensus_queue.cc:260] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.277657 30817 ts_tablet_manager.cc:1398] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 14:00:21.278033 30817 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:21.278177 30816 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:21.278482 30816 ts_tablet_manager.cc:1367] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 14:00:21.278474 30818 raft_consensus.cc:348] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.278714 30818 raft_consensus.cc:374] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.278802 30818 raft_consensus.cc:729] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.279212 30818 consensus_queue.cc:260] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.279825 30816 raft_consensus.cc:348] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.280121 30816 raft_consensus.cc:374] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.280246 30816 raft_consensus.cc:729] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.280642 30816 consensus_queue.cc:260] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.281400 30816 ts_tablet_manager.cc:1398] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:21.281558 30818 ts_tablet_manager.cc:1398] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:21.281858 30820 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:21.282053 30820 ts_tablet_manager.cc:1367] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20240501 14:00:21.282895 30820 raft_consensus.cc:348] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.283136 30820 raft_consensus.cc:374] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.283208 30820 raft_consensus.cc:729] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.283463 30820 consensus_queue.cc:260] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.284158 30820 ts_tablet_manager.cc:1398] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:21.285017 30817 tablet_bootstrap.cc:654] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.288719 30817 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:21.288849 30817 ts_tablet_manager.cc:1367] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240501 14:00:21.289500 30817 raft_consensus.cc:348] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.289659 30817 raft_consensus.cc:374] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.289723 30817 raft_consensus.cc:729] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.289902 30817 consensus_queue.cc:260] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.290354 30817 ts_tablet_manager.cc:1398] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:00:21.358434 30719 raft_consensus.cc:482] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.358621 30719 raft_consensus.cc:504] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.359285 30719 leader_election.cc:288] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:21.359776 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0a888dc37d44c69dea73f9ffd556f6" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:21.359792 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0a888dc37d44c69dea73f9ffd556f6" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:21.359992 29375 raft_consensus.cc:2454] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:21.360025 29507 raft_consensus.cc:2454] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:21.360440 29181 leader_election.cc:302] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:21.360693 30719 raft_consensus.cc:2781] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.360810 30719 raft_consensus.cc:482] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.360882 30719 raft_consensus.cc:3037] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.362821 30719 raft_consensus.cc:504] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.363358 30719 leader_election.cc:288] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:21.363806 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0a888dc37d44c69dea73f9ffd556f6" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:21.363819 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0a888dc37d44c69dea73f9ffd556f6" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:21.363997 29375 raft_consensus.cc:3037] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.364004 29507 raft_consensus.cc:3037] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.365933 29507 raft_consensus.cc:2454] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:21.365952 29375 raft_consensus.cc:2454] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:21.366333 29180 leader_election.cc:302] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:21.366631 30719 raft_consensus.cc:2781] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.366763 30719 raft_consensus.cc:686] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:21.367038 30719 consensus_queue.cc:237] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.369966 29105 catalog_manager.cc:5484] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:21.472911 30245 raft_consensus.cc:482] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.473096 30245 raft_consensus.cc:504] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.473750 30245 leader_election.cc:288] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:21.474258 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ce8f6f27b4ab19ff5a1bb972ca202" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:21.474242 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ce8f6f27b4ab19ff5a1bb972ca202" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:21.474459 29507 raft_consensus.cc:2454] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:21.474483 29246 raft_consensus.cc:2454] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:21.474896 29571 leader_election.cc:302] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:21.475173 30245 raft_consensus.cc:2781] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.475281 30245 raft_consensus.cc:482] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.475368 30245 raft_consensus.cc:3037] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.477169 30245 raft_consensus.cc:504] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.477739 30245 leader_election.cc:288] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:21.478124 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ce8f6f27b4ab19ff5a1bb972ca202" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:21.478114 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ce8f6f27b4ab19ff5a1bb972ca202" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:21.478324 29507 raft_consensus.cc:3037] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.478330 29246 raft_consensus.cc:3037] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.480135 29507 raft_consensus.cc:2454] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:21.480495 29571 leader_election.cc:302] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:21.480810 30245 raft_consensus.cc:2781] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.480954 29246 raft_consensus.cc:2454] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:21.481674 30245 raft_consensus.cc:686] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:21.481915 30830 raft_consensus.cc:482] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.482059 30830 raft_consensus.cc:504] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.482230 30245 consensus_queue.cc:237] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:21.482932 30830 leader_election.cc:288] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:21.483229 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:21.483250 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:21.483441 29376 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:21.483462 29507 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:21.483875 29571 leader_election.cc:302] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:21.484242 30830 raft_consensus.cc:2781] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.484377 30830 raft_consensus.cc:482] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.484486 30830 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.487380 30830 raft_consensus.cc:504] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.488426 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:21.488598 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:21.488693 29507 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.488795 29376 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.491644 29376 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:21.492039 29572 leader_election.cc:302] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:21.493446 30831 raft_consensus.cc:2781] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.494160 29507 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:21.495841 30831 raft_consensus.cc:686] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:21.495955 30830 leader_election.cc:288] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:21.496203 30831 consensus_queue.cc:237] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:21.498193 29105 catalog_manager.cc:5484] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:21.503861 29105 catalog_manager.cc:5484] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:21.766086 30650 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.771229 30719 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:21.860924 30836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:21.861302 30836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:21.869707 30836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:00:21.923719 30833 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.952124 30831 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.983376 30833 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.998658 30833 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:22.113571 29571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 64400.1us
W20240501 14:00:22.152031 29570 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 46867.5us
INSERT report
rows total: 2000
time total: 274.418 ms
time per row: 0.137209 ms
I20240501 14:00:22.296237 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:22.305428 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.305732 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.305963 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426bc72f14fb4490b329ab52bf7ca69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.306136 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426bc72f14fb4490b329ab52bf7ca69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.307212 29485 tablet_service.cc:1467] Processing CreateTablet for tablet b508eaf4280143df96ef3c72d54194ee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.307220 29616 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad2362aa0341d1b4faf9907d143458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.307612 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ad2362aa0341d1b4faf9907d143458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.307621 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b508eaf4280143df96ef3c72d54194ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.310662 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.311077 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426bc72f14fb4490b329ab52bf7ca69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.311955 29617 tablet_service.cc:1467] Processing CreateTablet for tablet b508eaf4280143df96ef3c72d54194ee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.312340 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b508eaf4280143df96ef3c72d54194ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.313370 29356 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad2362aa0341d1b4faf9907d143458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.313719 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ad2362aa0341d1b4faf9907d143458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.313824 29224 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad2362aa0341d1b4faf9907d143458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.314183 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ad2362aa0341d1b4faf9907d143458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.321311 30898 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:22.323346 30898 tablet_bootstrap.cc:654] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.325294 29226 tablet_service.cc:1467] Processing CreateTablet for tablet b508eaf4280143df96ef3c72d54194ee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.325796 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b508eaf4280143df96ef3c72d54194ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.331910 30899 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:22.333786 30899 tablet_bootstrap.cc:654] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.336177 30897 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:22.337661 30898 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:22.337893 30898 ts_tablet_manager.cc:1367] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20240501 14:00:22.338104 30899 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:22.338294 30899 ts_tablet_manager.cc:1367] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 14:00:22.338626 30897 tablet_bootstrap.cc:654] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.338791 30898 raft_consensus.cc:348] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.339069 30898 raft_consensus.cc:374] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.339074 30899 raft_consensus.cc:348] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.339202 30898 raft_consensus.cc:729] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.339327 30899 raft_consensus.cc:374] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.339497 30899 raft_consensus.cc:729] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.339464 30898 consensus_queue.cc:260] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.339872 30900 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:22.340270 30898 ts_tablet_manager.cc:1398] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:22.340615 30898 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:22.339761 30899 consensus_queue.cc:260] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.341288 30899 ts_tablet_manager.cc:1398] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:22.342094 30899 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:22.343829 30898 tablet_bootstrap.cc:654] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.344321 30899 tablet_bootstrap.cc:654] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.347417 30897 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:22.347631 30897 ts_tablet_manager.cc:1367] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240501 14:00:22.348537 30897 raft_consensus.cc:348] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.348870 30897 raft_consensus.cc:374] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.349035 30897 raft_consensus.cc:729] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.349340 30897 consensus_queue.cc:260] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.350924 30897 ts_tablet_manager.cc:1398] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:22.351270 30897 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:22.353906 30897 tablet_bootstrap.cc:654] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.353924 30898 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:22.354097 30898 ts_tablet_manager.cc:1367] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240501 14:00:22.355363 30898 raft_consensus.cc:348] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.355760 30898 raft_consensus.cc:374] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.355973 30898 raft_consensus.cc:729] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.356384 30898 consensus_queue.cc:260] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.358458 30898 ts_tablet_manager.cc:1398] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:22.358944 30898 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:22.361132 30898 tablet_bootstrap.cc:654] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.361593 30900 tablet_bootstrap.cc:654] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.369421 30900 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:22.369659 30900 ts_tablet_manager.cc:1367] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240501 14:00:22.370725 30900 raft_consensus.cc:348] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.371016 30900 raft_consensus.cc:374] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.371124 30900 raft_consensus.cc:729] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.371353 30898 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:22.371603 30898 ts_tablet_manager.cc:1367] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 14:00:22.371492 30900 consensus_queue.cc:260] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.372730 30898 raft_consensus.cc:348] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.373093 30898 raft_consensus.cc:374] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.373257 30898 raft_consensus.cc:729] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.373355 30900 ts_tablet_manager.cc:1398] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:22.373780 30900 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:22.373692 30898 consensus_queue.cc:260] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.375392 30898 ts_tablet_manager.cc:1398] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:22.377460 30897 tablet_bootstrap.cc:492] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:22.377655 30897 ts_tablet_manager.cc:1367] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.005s
I20240501 14:00:22.377970 30900 tablet_bootstrap.cc:654] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.378530 30897 raft_consensus.cc:348] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.378736 30897 raft_consensus.cc:374] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.378813 30897 raft_consensus.cc:729] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.379065 30897 consensus_queue.cc:260] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.379585 30899 tablet_bootstrap.cc:492] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:22.379635 30897 ts_tablet_manager.cc:1398] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:22.379751 30899 ts_tablet_manager.cc:1367] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.017s
I20240501 14:00:22.380712 30899 raft_consensus.cc:348] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.380887 30899 raft_consensus.cc:374] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.380967 30899 raft_consensus.cc:729] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.381238 30899 consensus_queue.cc:260] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.381695 30899 ts_tablet_manager.cc:1398] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:22.382546 30900 tablet_bootstrap.cc:492] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:22.382709 30900 ts_tablet_manager.cc:1367] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 14:00:22.383289 30900 raft_consensus.cc:348] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.383450 30900 raft_consensus.cc:374] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.383518 30900 raft_consensus.cc:729] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.383787 30900 consensus_queue.cc:260] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.385206 30900 ts_tablet_manager.cc:1398] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:00:22.429811 30791 raft_consensus.cc:482] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.430001 30791 raft_consensus.cc:504] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.430722 30791 leader_election.cc:288] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:22.431228 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426bc72f14fb4490b329ab52bf7ca69c" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:22.431443 29246 raft_consensus.cc:2454] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:22.431380 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426bc72f14fb4490b329ab52bf7ca69c" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:22.431682 29376 raft_consensus.cc:2454] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:22.431838 29440 leader_election.cc:302] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:22.432129 30791 raft_consensus.cc:2781] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.432251 30791 raft_consensus.cc:482] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.432339 30791 raft_consensus.cc:3037] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.434247 30791 raft_consensus.cc:504] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.434736 30791 leader_election.cc:288] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:22.434907 30569 raft_consensus.cc:482] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.435078 30569 raft_consensus.cc:504] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.435209 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426bc72f14fb4490b329ab52bf7ca69c" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:22.435284 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426bc72f14fb4490b329ab52bf7ca69c" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:22.435410 29246 raft_consensus.cc:3037] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.435474 29376 raft_consensus.cc:3037] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.435766 30569 leader_election.cc:288] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:22.436272 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad2362aa0341d1b4faf9907d143458" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:22.436410 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad2362aa0341d1b4faf9907d143458" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:22.436517 29244 raft_consensus.cc:2454] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:22.436619 29636 raft_consensus.cc:2454] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:22.436929 29310 leader_election.cc:302] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:22.437291 30569 raft_consensus.cc:2781] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.437398 30569 raft_consensus.cc:482] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.437498 30569 raft_consensus.cc:3037] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.438078 29246 raft_consensus.cc:2454] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:22.438249 29376 raft_consensus.cc:2454] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:22.438489 29440 leader_election.cc:302] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:22.438887 30791 raft_consensus.cc:2781] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.439275 30791 raft_consensus.cc:686] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:22.439594 30791 consensus_queue.cc:237] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.439927 30569 raft_consensus.cc:504] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.440414 30569 leader_election.cc:288] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:22.442127 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad2362aa0341d1b4faf9907d143458" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:22.442138 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad2362aa0341d1b4faf9907d143458" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:22.442296 29246 raft_consensus.cc:3037] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.442317 29636 raft_consensus.cc:3037] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.444185 29246 raft_consensus.cc:2454] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:22.444188 29636 raft_consensus.cc:2454] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:22.444577 29310 leader_election.cc:302] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:22.444847 30569 raft_consensus.cc:2781] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.445294 30569 raft_consensus.cc:686] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:22.445372 29099 catalog_manager.cc:5484] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:22.445806 30569 consensus_queue.cc:237] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:22.449095 29105 catalog_manager.cc:5484] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:22.629712 30791 raft_consensus.cc:482] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.629896 30791 raft_consensus.cc:504] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.630353 30791 leader_election.cc:288] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:22.630780 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b508eaf4280143df96ef3c72d54194ee" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:22.631007 29636 raft_consensus.cc:2454] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:22.630934 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b508eaf4280143df96ef3c72d54194ee" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:22.631155 29246 raft_consensus.cc:2454] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:22.631352 29440 leader_election.cc:302] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:22.631661 30791 raft_consensus.cc:2781] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.631744 30791 raft_consensus.cc:482] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.631865 30791 raft_consensus.cc:3037] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.633718 30791 raft_consensus.cc:504] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.634292 30791 leader_election.cc:288] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:22.634690 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b508eaf4280143df96ef3c72d54194ee" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:22.634836 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b508eaf4280143df96ef3c72d54194ee" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:22.634927 29636 raft_consensus.cc:3037] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.635047 29246 raft_consensus.cc:3037] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.637163 29636 raft_consensus.cc:2454] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:22.637281 29246 raft_consensus.cc:2454] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:22.637742 29440 leader_election.cc:302] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:22.637995 30791 raft_consensus.cc:2781] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.638200 30791 raft_consensus.cc:686] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:22.638516 30791 consensus_queue.cc:237] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:22.641211 29105 catalog_manager.cc:5484] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:22.855830 30791 consensus_queue.cc:1035] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.860936 30918 consensus_queue.cc:1035] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.919673 30569 consensus_queue.cc:1035] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.924458 30912 consensus_queue.cc:1035] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.046125 30915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:23.046427 30915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:23.054590 30915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:00:23.105804 30918 consensus_queue.cc:1035] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:23.114112 30910 consensus_queue.cc:1035] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.295111 29181 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 63022.6us
INSERT report
rows total: 2000
time total: 272.87 ms
time per row: 0.136435 ms
I20240501 14:00:23.476846 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:23.487108 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 49b72e2ac9df4426b071a7e5004e12e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.487308 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 49b72e2ac9df4426b071a7e5004e12e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.487622 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b72e2ac9df4426b071a7e5004e12e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.487744 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b72e2ac9df4426b071a7e5004e12e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.488780 29486 tablet_service.cc:1467] Processing CreateTablet for tablet a6e94a79c4514fe4a3ae8e41c7792dc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.489161 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e94a79c4514fe4a3ae8e41c7792dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.489785 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce342cbfad4419b3e2e8d3a52c4628 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.490164 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce342cbfad4419b3e2e8d3a52c4628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.493505 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce342cbfad4419b3e2e8d3a52c4628 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.493891 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce342cbfad4419b3e2e8d3a52c4628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.498431 29357 tablet_service.cc:1467] Processing CreateTablet for tablet a6e94a79c4514fe4a3ae8e41c7792dc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.498790 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e94a79c4514fe4a3ae8e41c7792dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.500355 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 49b72e2ac9df4426b071a7e5004e12e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.500767 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b72e2ac9df4426b071a7e5004e12e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.501771 29226 tablet_service.cc:1467] Processing CreateTablet for tablet a6e94a79c4514fe4a3ae8e41c7792dc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.502297 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e94a79c4514fe4a3ae8e41c7792dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.505484 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce342cbfad4419b3e2e8d3a52c4628 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.505928 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce342cbfad4419b3e2e8d3a52c4628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.515328 30978 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:23.516060 30977 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:23.519367 30978 tablet_bootstrap.cc:654] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.521906 30976 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:23.522485 30978 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:23.522728 30978 ts_tablet_manager.cc:1367] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 14:00:23.523708 30976 tablet_bootstrap.cc:654] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.523566 30978 raft_consensus.cc:348] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.523871 30977 tablet_bootstrap.cc:654] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.523892 30978 raft_consensus.cc:374] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.524061 30978 raft_consensus.cc:729] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.524398 30978 consensus_queue.cc:260] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.529225 30978 ts_tablet_manager.cc:1398] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:00:23.530592 30980 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:23.530969 30977 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:23.531114 30977 ts_tablet_manager.cc:1367] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240501 14:00:23.531934 30977 raft_consensus.cc:348] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.532137 30977 raft_consensus.cc:374] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.532229 30977 raft_consensus.cc:729] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.532435 30980 tablet_bootstrap.cc:654] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.532491 30977 consensus_queue.cc:260] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.533331 30977 ts_tablet_manager.cc:1398] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:23.533679 30977 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:23.536538 30977 tablet_bootstrap.cc:654] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.542176 30980 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:23.542343 30980 ts_tablet_manager.cc:1367] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 14:00:23.542797 30977 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:23.542958 30977 ts_tablet_manager.cc:1367] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 14:00:23.543226 30980 raft_consensus.cc:348] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.543426 30980 raft_consensus.cc:374] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.543524 30980 raft_consensus.cc:729] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.543709 30977 raft_consensus.cc:348] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.543751 30980 consensus_queue.cc:260] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.543922 30977 raft_consensus.cc:374] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.544005 30977 raft_consensus.cc:729] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.544327 30980 ts_tablet_manager.cc:1398] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:23.544281 30977 consensus_queue.cc:260] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.544862 30977 ts_tablet_manager.cc:1398] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:23.545017 30980 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:23.545303 30977 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:23.545647 30976 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:23.545850 30976 ts_tablet_manager.cc:1367] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.017s
I20240501 14:00:23.546860 30980 tablet_bootstrap.cc:654] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.546861 30977 tablet_bootstrap.cc:654] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.546711 30976 raft_consensus.cc:348] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.546947 30976 raft_consensus.cc:374] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.547066 30976 raft_consensus.cc:729] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.547374 30976 consensus_queue.cc:260] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.548802 30976 ts_tablet_manager.cc:1398] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:23.549162 30976 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:23.551160 30976 tablet_bootstrap.cc:654] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.556237 30980 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:23.556586 30980 ts_tablet_manager.cc:1367] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 14:00:23.557686 30980 raft_consensus.cc:348] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.558097 30980 raft_consensus.cc:374] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.558379 30980 raft_consensus.cc:729] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.558715 30980 consensus_queue.cc:260] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.560464 30980 ts_tablet_manager.cc:1398] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:23.562079 30977 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:23.562268 30977 ts_tablet_manager.cc:1367] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20240501 14:00:23.563107 30977 raft_consensus.cc:348] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.563306 30977 raft_consensus.cc:374] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.563356 30976 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:23.563411 30977 raft_consensus.cc:729] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.563529 30976 ts_tablet_manager.cc:1367] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240501 14:00:23.563658 30977 consensus_queue.cc:260] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.564229 30977 ts_tablet_manager.cc:1398] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:23.564215 30976 raft_consensus.cc:348] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.564412 30976 raft_consensus.cc:374] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.564522 30976 raft_consensus.cc:729] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.564765 30976 consensus_queue.cc:260] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.565275 30976 ts_tablet_manager.cc:1398] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:23.566044 30976 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:23.567281 30976 tablet_bootstrap.cc:654] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.569492 30976 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:23.569626 30976 ts_tablet_manager.cc:1367] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:23.570295 30976 raft_consensus.cc:348] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.570493 30976 raft_consensus.cc:374] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.570591 30976 raft_consensus.cc:729] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.570784 30976 consensus_queue.cc:260] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.571281 30976 ts_tablet_manager.cc:1398] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:23.587968 30912 raft_consensus.cc:482] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.588174 30912 raft_consensus.cc:504] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.588872 30912 leader_election.cc:288] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:23.589417 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e94a79c4514fe4a3ae8e41c7792dc9" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:23.589481 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e94a79c4514fe4a3ae8e41c7792dc9" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:23.589643 29246 raft_consensus.cc:2454] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:23.589725 29506 raft_consensus.cc:2454] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:23.590072 29310 leader_election.cc:302] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:23.590349 30912 raft_consensus.cc:2781] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.590446 30912 raft_consensus.cc:482] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.590548 30912 raft_consensus.cc:3037] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.592752 30912 raft_consensus.cc:504] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.593472 30912 leader_election.cc:288] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:23.593866 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e94a79c4514fe4a3ae8e41c7792dc9" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:23.593895 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e94a79c4514fe4a3ae8e41c7792dc9" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:23.594075 29246 raft_consensus.cc:3037] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.594115 29506 raft_consensus.cc:3037] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.596331 29506 raft_consensus.cc:2454] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:23.596345 29246 raft_consensus.cc:2454] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:23.596771 29310 leader_election.cc:302] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:23.597131 30912 raft_consensus.cc:2781] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.597637 30912 raft_consensus.cc:686] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:23.597882 30912 consensus_queue.cc:237] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:23.601255 29102 catalog_manager.cc:5484] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:23.655472 30918 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.655676 30918 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.656376 30918 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:23.656869 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:23.656909 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:23.657137 29375 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:23.657191 29636 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:23.657558 29442 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:23.657824 30910 raft_consensus.cc:2781] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.657930 30910 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.658018 30910 raft_consensus.cc:3037] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.660054 30910 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.660652 30910 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:23.661167 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:23.661193 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:23.661409 29636 raft_consensus.cc:3037] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.661432 29375 raft_consensus.cc:3037] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.663290 29375 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:23.663303 29636 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:23.663765 29442 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:23.664047 30910 raft_consensus.cc:2781] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.664173 30910 raft_consensus.cc:686] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:23.664490 30910 consensus_queue.cc:237] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:23.667522 29102 catalog_manager.cc:5484] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:23.830605 30910 raft_consensus.cc:482] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.830837 30910 raft_consensus.cc:504] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.831429 30910 leader_election.cc:288] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:23.831930 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:23.832016 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:23.832118 29375 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:23.832278 29636 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:23.832484 29442 leader_election.cc:302] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:23.832727 30910 raft_consensus.cc:2781] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.832816 30910 raft_consensus.cc:482] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.832897 30910 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.834684 30910 raft_consensus.cc:504] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.835193 30910 leader_election.cc:288] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:23.835556 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:23.835664 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:23.835749 29636 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.835847 29375 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.837635 29375 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:23.837844 29636 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:23.838012 29442 leader_election.cc:302] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:23.838294 30910 raft_consensus.cc:2781] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.838459 30910 raft_consensus.cc:686] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:23.838701 30910 consensus_queue.cc:237] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:23.841466 29102 catalog_manager.cc:5484] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:23.996150 30912 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.002878 30569 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.167199 30910 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.171533 30918 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:24.238768 30992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:24.239128 30992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:24.247253 30992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:00:24.366220 29376 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:24.369853 29637 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:24.370735 30918 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.393503 31032 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:24.427534 29442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 57180.2us
INSERT report
rows total: 2000
time total: 294.05 ms
time per row: 0.147025 ms
I20240501 14:00:24.685448 29102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:24.693733 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.694234 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.694392 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.694938 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.695693 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 8227f23eac39422eb5fa519d202a0d57 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.696101 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8227f23eac39422eb5fa519d202a0d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.697527 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.697916 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.700024 31055 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:24.700332 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 954bd347d6944d9cbc7587cb2435487e (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.700716 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954bd347d6944d9cbc7587cb2435487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.701925 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 954bd347d6944d9cbc7587cb2435487e (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.702136 31055 tablet_bootstrap.cc:654] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.702314 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954bd347d6944d9cbc7587cb2435487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.703744 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 8227f23eac39422eb5fa519d202a0d57 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.704144 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8227f23eac39422eb5fa519d202a0d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.705272 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 8227f23eac39422eb5fa519d202a0d57 (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.705513 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8227f23eac39422eb5fa519d202a0d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.707363 31057 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:24.708668 31055 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:24.708767 31055 ts_tablet_manager.cc:1367] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:00:24.709604 31055 raft_consensus.cc:348] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.709821 31055 raft_consensus.cc:374] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.709910 31055 raft_consensus.cc:729] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.710063 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 954bd347d6944d9cbc7587cb2435487e (DEFAULT_TABLE table=rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.710211 31055 consensus_queue.cc:260] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.710512 31057 tablet_bootstrap.cc:654] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.710530 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954bd347d6944d9cbc7587cb2435487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.710978 31055 ts_tablet_manager.cc:1398] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:24.711280 31055 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:24.717742 31055 tablet_bootstrap.cc:654] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.719256 31060 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:24.722482 31059 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:24.723138 31060 tablet_bootstrap.cc:654] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.726286 31057 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:24.726361 31055 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:24.726490 31059 tablet_bootstrap.cc:654] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.726511 31057 ts_tablet_manager.cc:1367] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20240501 14:00:24.726521 31055 ts_tablet_manager.cc:1367] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240501 14:00:24.727227 31055 raft_consensus.cc:348] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.727237 31057 raft_consensus.cc:348] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.727443 31057 raft_consensus.cc:374] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.727459 31055 raft_consensus.cc:374] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.727530 31055 raft_consensus.cc:729] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.727614 31057 raft_consensus.cc:729] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.727802 31055 consensus_queue.cc:260] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.727871 31057 consensus_queue.cc:260] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.728474 31055 ts_tablet_manager.cc:1398] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:24.729549 31057 ts_tablet_manager.cc:1398] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:24.730350 31057 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:24.730780 31059 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:24.730957 31059 ts_tablet_manager.cc:1367] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240501 14:00:24.729935 31060 tablet_bootstrap.cc:492] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:24.731549 31060 ts_tablet_manager.cc:1367] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 14:00:24.732371 31060 raft_consensus.cc:348] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.732606 31060 raft_consensus.cc:374] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.732585 31059 raft_consensus.cc:348] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.732708 31060 raft_consensus.cc:729] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.732803 31059 raft_consensus.cc:374] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.732921 31059 raft_consensus.cc:729] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.732949 31060 consensus_queue.cc:260] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.733219 31059 consensus_queue.cc:260] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.733525 31060 ts_tablet_manager.cc:1398] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:24.734159 31057 tablet_bootstrap.cc:654] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.736033 31059 ts_tablet_manager.cc:1398] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:24.736501 31059 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:24.738175 31057 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:24.738338 31057 ts_tablet_manager.cc:1367] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 14:00:24.738641 31059 tablet_bootstrap.cc:654] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.739105 31057 raft_consensus.cc:348] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.739423 31057 raft_consensus.cc:374] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.739509 31057 raft_consensus.cc:729] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.739820 31057 consensus_queue.cc:260] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.740453 31057 ts_tablet_manager.cc:1398] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:24.741217 31059 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:24.741227 31057 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:24.741444 31059 ts_tablet_manager.cc:1367] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 14:00:24.742218 31059 raft_consensus.cc:348] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.742517 31059 raft_consensus.cc:374] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.742612 31059 raft_consensus.cc:729] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.742870 31059 consensus_queue.cc:260] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.743422 31059 ts_tablet_manager.cc:1398] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:24.743728 31059 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:24.745421 31059 tablet_bootstrap.cc:654] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.746330 31057 tablet_bootstrap.cc:654] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.747581 31059 tablet_bootstrap.cc:492] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:24.747712 31059 ts_tablet_manager.cc:1367] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:24.748184 31057 tablet_bootstrap.cc:492] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:24.748342 31057 ts_tablet_manager.cc:1367] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240501 14:00:24.748440 31059 raft_consensus.cc:348] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.748656 31059 raft_consensus.cc:374] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.748747 31059 raft_consensus.cc:729] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.748996 31059 consensus_queue.cc:260] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.749233 31057 raft_consensus.cc:348] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.749482 31057 raft_consensus.cc:374] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.749553 31059 ts_tablet_manager.cc:1398] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:24.749626 31057 raft_consensus.cc:729] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.749856 31057 consensus_queue.cc:260] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.750304 31057 ts_tablet_manager.cc:1398] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:24.812949 30719 raft_consensus.cc:482] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.813151 30719 raft_consensus.cc:504] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.813673 30719 leader_election.cc:288] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:24.814105 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954bd347d6944d9cbc7587cb2435487e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:24.814121 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954bd347d6944d9cbc7587cb2435487e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:24.814333 29506 raft_consensus.cc:2454] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:24.814325 29637 raft_consensus.cc:2454] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:24.814738 29180 leader_election.cc:302] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:24.814980 30719 raft_consensus.cc:2781] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.815078 30719 raft_consensus.cc:482] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.815168 30719 raft_consensus.cc:3037] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.817291 30719 raft_consensus.cc:504] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.817835 30719 leader_election.cc:288] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:24.818205 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954bd347d6944d9cbc7587cb2435487e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:24.818281 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954bd347d6944d9cbc7587cb2435487e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:24.818403 29637 raft_consensus.cc:3037] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.818475 29506 raft_consensus.cc:3037] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.820252 29506 raft_consensus.cc:2454] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:24.820309 29637 raft_consensus.cc:2454] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:24.820672 29180 leader_election.cc:302] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:24.820962 30719 raft_consensus.cc:2781] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.821120 30719 raft_consensus.cc:686] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:24.821334 30719 consensus_queue.cc:237] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:24.824321 29105 catalog_manager.cc:5484] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:24.941625 30719 raft_consensus.cc:482] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.941802 30719 raft_consensus.cc:504] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.942447 30719 leader_election.cc:288] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:24.942914 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ebbac3d0aa4c23b42157d5c0d2acc9" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:24.942983 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ebbac3d0aa4c23b42157d5c0d2acc9" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:24.943164 29377 raft_consensus.cc:2454] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:24.943207 29637 raft_consensus.cc:2454] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:24.943640 29181 leader_election.cc:302] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:24.943935 30719 raft_consensus.cc:2781] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.944042 30719 raft_consensus.cc:482] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.944137 30719 raft_consensus.cc:3037] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.947044 30719 raft_consensus.cc:504] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.947742 30719 leader_election.cc:288] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:24.948091 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ebbac3d0aa4c23b42157d5c0d2acc9" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:24.948153 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ebbac3d0aa4c23b42157d5c0d2acc9" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:24.948285 29377 raft_consensus.cc:3037] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.948433 29637 raft_consensus.cc:3037] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.950433 29377 raft_consensus.cc:2454] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:24.950968 29181 leader_election.cc:302] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:24.951232 29637 raft_consensus.cc:2454] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:24.951531 31042 raft_consensus.cc:2781] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.951690 31042 raft_consensus.cc:686] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:24.951944 31042 consensus_queue.cc:237] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.956374 29100 catalog_manager.cc:5484] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:24.958766 31050 raft_consensus.cc:482] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.958914 31050 raft_consensus.cc:504] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.959604 31050 leader_election.cc:288] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:24.960048 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8227f23eac39422eb5fa519d202a0d57" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:24.960073 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8227f23eac39422eb5fa519d202a0d57" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:24.960249 29246 raft_consensus.cc:2454] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:24.960299 29637 raft_consensus.cc:2454] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:24.960752 29310 leader_election.cc:302] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:24.961009 31050 raft_consensus.cc:2781] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.961126 31050 raft_consensus.cc:482] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.961220 31050 raft_consensus.cc:3037] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.963068 31050 raft_consensus.cc:504] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.963569 31050 leader_election.cc:288] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:24.964041 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8227f23eac39422eb5fa519d202a0d57" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:24.964076 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8227f23eac39422eb5fa519d202a0d57" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:24.964303 29246 raft_consensus.cc:3037] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.964341 29637 raft_consensus.cc:3037] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.967162 29246 raft_consensus.cc:2454] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:24.967151 29637 raft_consensus.cc:2454] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:24.967620 29310 leader_election.cc:302] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:24.967928 31050 raft_consensus.cc:2781] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.968114 31050 raft_consensus.cc:686] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:24.968416 31050 consensus_queue.cc:237] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:24.971793 29100 catalog_manager.cc:5484] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:25.228055 30719 consensus_queue.cc:1035] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:25.234203 31042 consensus_queue.cc:1035] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:25.334940 30719 consensus_queue.cc:1035] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:25.338984 30719 consensus_queue.cc:1035] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:25.370311 31072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:25.370612 31072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:25.378160 31072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 14:00:25.460934 31050 consensus_queue.cc:1035] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:25.507889 30912 consensus_queue.cc:1035] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.405 ms
time per row: 0.114202 ms
I20240501 14:00:25.767915 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:25.777789 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffd2100f104b339bfdc2d272ba3729 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.777801 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffd2100f104b339bfdc2d272ba3729 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.777832 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 83ca18280bbb49058e5aa80a3832b666 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.778261 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffd2100f104b339bfdc2d272ba3729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.778261 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffd2100f104b339bfdc2d272ba3729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.778280 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ca18280bbb49058e5aa80a3832b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.779053 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffd2100f104b339bfdc2d272ba3729 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.779443 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffd2100f104b339bfdc2d272ba3729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.781780 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 83ca18280bbb49058e5aa80a3832b666 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.782194 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ca18280bbb49058e5aa80a3832b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.782159 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 83ca18280bbb49058e5aa80a3832b666 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.782560 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ca18280bbb49058e5aa80a3832b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.785439 29486 tablet_service.cc:1467] Processing CreateTablet for tablet 241a2fdfe954476ba9bd2cfeb4b931ac (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.785802 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 241a2fdfe954476ba9bd2cfeb4b931ac (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.785962 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241a2fdfe954476ba9bd2cfeb4b931ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.786192 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241a2fdfe954476ba9bd2cfeb4b931ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.790231 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 241a2fdfe954476ba9bd2cfeb4b931ac (DEFAULT_TABLE table=rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.790822 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241a2fdfe954476ba9bd2cfeb4b931ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.795161 31133 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:25.798296 31133 tablet_bootstrap.cc:654] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.807294 31134 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:25.811331 31132 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:25.813824 31134 tablet_bootstrap.cc:654] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.815680 31132 tablet_bootstrap.cc:654] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.823277 31136 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:25.825959 31136 tablet_bootstrap.cc:654] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.828096 31132 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:25.828296 31132 ts_tablet_manager.cc:1367] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20240501 14:00:25.828593 31134 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:25.828856 31134 ts_tablet_manager.cc:1367] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.005s
I20240501 14:00:25.829233 31133 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:25.829285 31136 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:25.829213 31132 raft_consensus.cc:348] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.829404 31133 ts_tablet_manager.cc:1367] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.017s
I20240501 14:00:25.829509 31136 ts_tablet_manager.cc:1367] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:00:25.829553 31132 raft_consensus.cc:374] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.829697 31132 raft_consensus.cc:729] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.829906 31134 raft_consensus.cc:348] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.830178 31134 raft_consensus.cc:374] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.830050 31132 consensus_queue.cc:260] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.830179 31133 raft_consensus.cc:348] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.830312 31134 raft_consensus.cc:729] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.830389 31133 raft_consensus.cc:374] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.830343 31136 raft_consensus.cc:348] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.830487 31133 raft_consensus.cc:729] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.830595 31136 raft_consensus.cc:374] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.830765 31136 raft_consensus.cc:729] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.830863 31133 consensus_queue.cc:260] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.831030 31132 ts_tablet_manager.cc:1398] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:25.831024 31136 consensus_queue.cc:260] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.831327 31132 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:25.831446 31133 ts_tablet_manager.cc:1398] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:25.831768 31133 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:25.832736 31136 ts_tablet_manager.cc:1398] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:25.833113 31136 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:25.833703 31133 tablet_bootstrap.cc:654] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.830549 31134 consensus_queue.cc:260] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.834820 31136 tablet_bootstrap.cc:654] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.835085 31134 ts_tablet_manager.cc:1398] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:25.835513 31134 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:25.837697 31134 tablet_bootstrap.cc:654] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.840250 31133 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:25.840404 31132 tablet_bootstrap.cc:654] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.840435 31133 ts_tablet_manager.cc:1367] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240501 14:00:25.841447 31133 raft_consensus.cc:348] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.841701 31133 raft_consensus.cc:374] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.841782 31133 raft_consensus.cc:729] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.842070 31133 consensus_queue.cc:260] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.843555 31133 ts_tablet_manager.cc:1398] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:25.845454 31136 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:25.845666 31136 ts_tablet_manager.cc:1367] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240501 14:00:25.847546 31136 raft_consensus.cc:348] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.847783 31136 raft_consensus.cc:374] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.847882 31136 raft_consensus.cc:729] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.848143 31136 consensus_queue.cc:260] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.848724 31136 ts_tablet_manager.cc:1398] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:25.849134 31136 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:25.850950 31132 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:25.851173 31136 tablet_bootstrap.cc:654] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.851284 31132 ts_tablet_manager.cc:1367] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240501 14:00:25.853145 31134 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:25.853323 31134 ts_tablet_manager.cc:1367] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20240501 14:00:25.853350 31132 raft_consensus.cc:348] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.853593 31132 raft_consensus.cc:374] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.853663 31132 raft_consensus.cc:729] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.853751 31136 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:25.853886 31136 ts_tablet_manager.cc:1367] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:25.853899 31132 consensus_queue.cc:260] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.854125 31134 raft_consensus.cc:348] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.854329 31134 raft_consensus.cc:374] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.854399 31134 raft_consensus.cc:729] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.854657 31136 raft_consensus.cc:348] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.854640 31134 consensus_queue.cc:260] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.854796 31136 raft_consensus.cc:374] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.854852 31136 raft_consensus.cc:729] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.855062 31136 consensus_queue.cc:260] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:25.855201 31134 ts_tablet_manager.cc:1398] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:25.855619 31136 ts_tablet_manager.cc:1398] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:25.856297 31132 ts_tablet_manager.cc:1398] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:25.898258 31040 raft_consensus.cc:482] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:25.898478 31040 raft_consensus.cc:504] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.899168 31040 leader_election.cc:288] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:25.899741 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241a2fdfe954476ba9bd2cfeb4b931ac" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:25.899744 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241a2fdfe954476ba9bd2cfeb4b931ac" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:25.899966 29506 raft_consensus.cc:2454] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:25.899968 29376 raft_consensus.cc:2454] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:25.900418 29572 leader_election.cc:302] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:25.900714 31040 raft_consensus.cc:2781] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:25.900847 31040 raft_consensus.cc:482] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:25.900949 31040 raft_consensus.cc:3037] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.901516 31120 raft_consensus.cc:482] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:25.901679 31120 raft_consensus.cc:504] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.902535 31120 leader_election.cc:288] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:25.902972 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ca18280bbb49058e5aa80a3832b666" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:25.903247 29637 raft_consensus.cc:2454] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:25.903180 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ca18280bbb49058e5aa80a3832b666" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:25.903390 29506 raft_consensus.cc:2454] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:25.903694 31040 raft_consensus.cc:504] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.903723 29179 leader_election.cc:302] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:25.903983 31120 raft_consensus.cc:2781] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:25.904099 31120 raft_consensus.cc:482] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:25.904198 31120 raft_consensus.cc:3037] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.904307 31040 leader_election.cc:288] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:25.904681 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241a2fdfe954476ba9bd2cfeb4b931ac" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:25.904868 29506 raft_consensus.cc:3037] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.907058 31120 raft_consensus.cc:504] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.907599 29506 raft_consensus.cc:2454] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:25.907902 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241a2fdfe954476ba9bd2cfeb4b931ac" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:25.908104 29376 raft_consensus.cc:3037] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.910101 29376 raft_consensus.cc:2454] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:25.910135 29571 leader_election.cc:302] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:25.910473 31120 leader_election.cc:288] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:25.910962 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ca18280bbb49058e5aa80a3832b666" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:25.911211 29506 raft_consensus.cc:3037] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.911195 31040 raft_consensus.cc:2781] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:25.911479 31040 raft_consensus.cc:686] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:25.911216 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ca18280bbb49058e5aa80a3832b666" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:25.912139 29637 raft_consensus.cc:3037] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.912297 31040 consensus_queue.cc:237] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:25.914566 29506 raft_consensus.cc:2454] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:25.914917 29637 raft_consensus.cc:2454] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:25.915087 29180 leader_election.cc:302] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:25.915412 31120 raft_consensus.cc:2781] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:25.915568 31120 raft_consensus.cc:686] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:25.915851 31120 consensus_queue.cc:237] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:25.916679 29102 catalog_manager.cc:5484] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:25.920197 29105 catalog_manager.cc:5484] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:26.049213 31032 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:26.049376 31032 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:26.050007 31032 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:26.050534 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:26.050534 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:26.050786 29246 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:26.050769 29376 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:26.051277 29440 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:26.051548 31032 raft_consensus.cc:2781] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:26.051664 31032 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:26.051753 31032 raft_consensus.cc:3037] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:26.053640 31032 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:26.054157 31032 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:26.054734 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:26.054752 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:26.054909 29246 raft_consensus.cc:3037] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:26.054944 29376 raft_consensus.cc:3037] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:26.056767 29246 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:26.057058 29376 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:26.057147 29440 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:26.057425 31032 raft_consensus.cc:2781] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:26.057559 31032 raft_consensus.cc:686] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:26.057811 31032 consensus_queue.cc:237] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:26.061287 29105 catalog_manager.cc:5484] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:26.439786 31110 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.442868 31122 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.448726 31040 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.454070 31120 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:26.477880 31148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:26.478448 31148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:26.488399 31148 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 14:00:26.587888 31032 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.694331 29375 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:26.695377 31204 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 244.083 ms
time per row: 0.122041 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:27.849865 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:27.859632 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.859760 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.860149 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b76fa6c27c44d7588cd0fb133113bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.860149 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b76fa6c27c44d7588cd0fb133113bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.860235 29617 tablet_service.cc:1467] Processing CreateTablet for tablet c451da0cce1447d2aa3209a8837b82a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.860235 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.860673 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b76fa6c27c44d7588cd0fb133113bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.860683 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c451da0cce1447d2aa3209a8837b82a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.861595 29356 tablet_service.cc:1467] Processing CreateTablet for tablet c451da0cce1447d2aa3209a8837b82a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.861917 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c451da0cce1447d2aa3209a8837b82a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.866040 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.866464 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.868176 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.868379 29226 tablet_service.cc:1467] Processing CreateTablet for tablet c451da0cce1447d2aa3209a8837b82a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.868579 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.868834 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c451da0cce1447d2aa3209a8837b82a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.873395 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (DEFAULT_TABLE table=rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.873837 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.883466 31221 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:27.885318 31222 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:27.886289 31220 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:27.887382 31222 tablet_bootstrap.cc:654] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.887586 31221 tablet_bootstrap.cc:654] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.888000 31220 tablet_bootstrap.cc:654] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.893088 31221 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:27.893366 31221 ts_tablet_manager.cc:1367] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 14:00:27.894290 31221 raft_consensus.cc:348] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.894650 31221 raft_consensus.cc:374] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.894848 31221 raft_consensus.cc:729] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.895181 31221 consensus_queue.cc:260] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.895843 31223 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:27.896306 31221 ts_tablet_manager.cc:1398] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:27.896732 31220 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:27.896910 31220 ts_tablet_manager.cc:1367] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 14:00:27.897846 31223 tablet_bootstrap.cc:654] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.897738 31220 raft_consensus.cc:348] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.897971 31220 raft_consensus.cc:374] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.898080 31220 raft_consensus.cc:729] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.898342 31220 consensus_queue.cc:260] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.899046 31220 ts_tablet_manager.cc:1398] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:27.899453 31220 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:27.896624 31221 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:27.901647 31220 tablet_bootstrap.cc:654] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.905814 31221 tablet_bootstrap.cc:654] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.907927 31220 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:27.908072 31222 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:27.908114 31220 ts_tablet_manager.cc:1367] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240501 14:00:27.908229 31222 ts_tablet_manager.cc:1367] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.009s
I20240501 14:00:27.908951 31222 raft_consensus.cc:348] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.909308 31222 raft_consensus.cc:374] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.909418 31222 raft_consensus.cc:729] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.909729 31223 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:27.909667 31222 consensus_queue.cc:260] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.909889 31223 ts_tablet_manager.cc:1367] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240501 14:00:27.910308 31222 ts_tablet_manager.cc:1398] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:27.910584 31222 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:27.910713 31223 raft_consensus.cc:348] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.910900 31223 raft_consensus.cc:374] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.910981 31223 raft_consensus.cc:729] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.911252 31221 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:27.911198 31223 consensus_queue.cc:260] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.911420 31221 ts_tablet_manager.cc:1367] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240501 14:00:27.911756 31223 ts_tablet_manager.cc:1398] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:27.912047 31223 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:27.912249 31221 raft_consensus.cc:348] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.912457 31221 raft_consensus.cc:374] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.912536 31221 raft_consensus.cc:729] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.912775 31222 tablet_bootstrap.cc:654] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.912773 31221 consensus_queue.cc:260] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.913447 31221 ts_tablet_manager.cc:1398] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:27.913734 31223 tablet_bootstrap.cc:654] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.908881 31220 raft_consensus.cc:348] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.915104 31220 raft_consensus.cc:374] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.915225 31220 raft_consensus.cc:729] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.915539 31220 consensus_queue.cc:260] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.916182 31220 ts_tablet_manager.cc:1398] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:00:27.917253 31223 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:27.917445 31223 ts_tablet_manager.cc:1367] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:27.918467 31223 raft_consensus.cc:348] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.918874 31223 raft_consensus.cc:374] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.919016 31223 raft_consensus.cc:729] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.919423 31223 consensus_queue.cc:260] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.920325 31223 ts_tablet_manager.cc:1398] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:27.921561 31222 tablet_bootstrap.cc:492] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:27.921869 31222 ts_tablet_manager.cc:1367] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240501 14:00:27.922994 31222 raft_consensus.cc:348] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.923413 31222 raft_consensus.cc:374] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.923560 31222 raft_consensus.cc:729] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.923947 31222 consensus_queue.cc:260] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:27.925401 31222 ts_tablet_manager.cc:1398] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:27.925864 31222 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:27.927563 31222 tablet_bootstrap.cc:654] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.930636 31222 tablet_bootstrap.cc:492] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:27.930781 31222 ts_tablet_manager.cc:1367] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:00:27.931475 31222 raft_consensus.cc:348] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.931676 31222 raft_consensus.cc:374] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.931771 31222 raft_consensus.cc:729] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.932004 31222 consensus_queue.cc:260] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:27.932605 31222 ts_tablet_manager.cc:1398] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:28.078995 31233 raft_consensus.cc:482] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.079185 31233 raft_consensus.cc:504] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.079965 31233 leader_election.cc:288] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:28.080453 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c451da0cce1447d2aa3209a8837b82a4" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:28.080518 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c451da0cce1447d2aa3209a8837b82a4" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:28.080657 29635 raft_consensus.cc:2454] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:28.080778 29375 raft_consensus.cc:2454] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:28.081159 29179 leader_election.cc:302] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:28.081465 31233 raft_consensus.cc:2781] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:28.081580 31233 raft_consensus.cc:482] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:28.081665 31233 raft_consensus.cc:3037] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.083743 31233 raft_consensus.cc:504] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.084324 31233 leader_election.cc:288] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:28.084889 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c451da0cce1447d2aa3209a8837b82a4" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:28.085146 29375 raft_consensus.cc:3037] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.085494 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c451da0cce1447d2aa3209a8837b82a4" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:28.085683 29635 raft_consensus.cc:3037] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.087363 29375 raft_consensus.cc:2454] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:28.087637 29635 raft_consensus.cc:2454] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:28.087762 29181 leader_election.cc:302] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:28.088146 31233 raft_consensus.cc:2781] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:28.088557 31233 raft_consensus.cc:686] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:28.088861 31233 consensus_queue.cc:237] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.092288 29099 catalog_manager.cc:5484] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:28.180920 31215 raft_consensus.cc:482] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.181144 31215 raft_consensus.cc:504] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.181752 31215 leader_election.cc:288] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:28.182224 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b76fa6c27c44d7588cd0fb133113bfe" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:28.182235 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b76fa6c27c44d7588cd0fb133113bfe" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:28.182432 29246 raft_consensus.cc:2454] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:28.182446 29506 raft_consensus.cc:2454] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:28.182878 29310 leader_election.cc:302] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:28.183125 31215 raft_consensus.cc:2781] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:28.183220 31215 raft_consensus.cc:482] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:28.183321 31215 raft_consensus.cc:3037] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.183988 31233 raft_consensus.cc:482] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.184166 31233 raft_consensus.cc:504] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:28.184875 31233 leader_election.cc:288] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:28.185299 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c1d1f40c8f4eaca06a3dc9c0304dce" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:28.185492 29506 raft_consensus.cc:2454] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:28.185505 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c1d1f40c8f4eaca06a3dc9c0304dce" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:28.185500 31215 raft_consensus.cc:504] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.185740 29635 raft_consensus.cc:2454] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:28.185897 29180 leader_election.cc:302] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:28.186177 31233 raft_consensus.cc:2781] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:28.186316 31233 raft_consensus.cc:482] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:28.186429 31215 leader_election.cc:288] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:28.186497 31233 raft_consensus.cc:3037] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.186755 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b76fa6c27c44d7588cd0fb133113bfe" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:28.186936 29246 raft_consensus.cc:3037] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.188691 31233 raft_consensus.cc:504] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:28.189249 31233 leader_election.cc:288] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:28.189476 29246 raft_consensus.cc:2454] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:28.189733 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c1d1f40c8f4eaca06a3dc9c0304dce" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:28.189931 29506 raft_consensus.cc:3037] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.189901 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c1d1f40c8f4eaca06a3dc9c0304dce" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:28.189982 29310 leader_election.cc:302] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:28.190165 29635 raft_consensus.cc:3037] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.190296 31215 raft_consensus.cc:2781] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:28.190543 31215 raft_consensus.cc:686] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:28.190866 31215 consensus_queue.cc:237] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:28.192432 29635 raft_consensus.cc:2454] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:28.192512 29506 raft_consensus.cc:2454] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:28.192948 29179 leader_election.cc:302] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:28.193117 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b76fa6c27c44d7588cd0fb133113bfe" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:28.193324 29507 raft_consensus.cc:3037] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.193622 31233 raft_consensus.cc:2781] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:28.193789 31233 raft_consensus.cc:686] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:28.194010 31233 consensus_queue.cc:237] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:28.194725 29099 catalog_manager.cc:5484] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:28.195674 29507 raft_consensus.cc:2454] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:28.198014 29105 catalog_manager.cc:5484] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:28.485741 31233 consensus_queue.cc:1035] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.517400 31244 consensus_queue.cc:1035] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.605687 31244 consensus_queue.cc:1035] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.611423 31232 consensus_queue.cc:1035] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:28.619261 31239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:28.619593 31239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:28.627148 31239 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 14:00:28.700273 31215 consensus_queue.cc:1035] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.705545 31215 consensus_queue.cc:1035] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.212 ms
time per row: 0.115106 ms
I20240501 14:00:29.007196 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:29.017261 29485 tablet_service.cc:1467] Processing CreateTablet for tablet e513634d27c5467fb49c9717c43904cd (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.017477 29355 tablet_service.cc:1467] Processing CreateTablet for tablet e513634d27c5467fb49c9717c43904cd (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.017741 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e513634d27c5467fb49c9717c43904cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.017853 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e513634d27c5467fb49c9717c43904cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.018749 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 192a6cc308da4a448f03717b6a439094 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.019201 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192a6cc308da4a448f03717b6a439094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.021371 29486 tablet_service.cc:1467] Processing CreateTablet for tablet c2c63489706e4d068adde1872f451ad2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.021728 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c63489706e4d068adde1872f451ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.022092 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 192a6cc308da4a448f03717b6a439094 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.022532 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192a6cc308da4a448f03717b6a439094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.025354 29617 tablet_service.cc:1467] Processing CreateTablet for tablet e513634d27c5467fb49c9717c43904cd (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.025385 29226 tablet_service.cc:1467] Processing CreateTablet for tablet c2c63489706e4d068adde1872f451ad2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.025751 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e513634d27c5467fb49c9717c43904cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.025794 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c63489706e4d068adde1872f451ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.033319 29357 tablet_service.cc:1467] Processing CreateTablet for tablet c2c63489706e4d068adde1872f451ad2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.033747 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c63489706e4d068adde1872f451ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.037236 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 192a6cc308da4a448f03717b6a439094 (DEFAULT_TABLE table=rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:29.037569 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192a6cc308da4a448f03717b6a439094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:29.041589 31300 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:29.044458 31300 tablet_bootstrap.cc:654] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.044629 31302 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:29.046486 31302 tablet_bootstrap.cc:654] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.086932 31305 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:29.088137 31306 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:29.090874 31306 tablet_bootstrap.cc:654] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:29.092272 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 48907.8us
I20240501 14:00:29.095054 31305 tablet_bootstrap.cc:654] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.096094 31306 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:29.096196 31302 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:29.096284 31306 ts_tablet_manager.cc:1367] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240501 14:00:29.096334 31302 ts_tablet_manager.cc:1367] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.028s
I20240501 14:00:29.096400 31300 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:29.096513 31300 ts_tablet_manager.cc:1367] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.055s user 0.020s sys 0.024s
I20240501 14:00:29.097138 31306 raft_consensus.cc:348] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.097081 31302 raft_consensus.cc:348] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.097143 31300 raft_consensus.cc:348] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.097324 31306 raft_consensus.cc:374] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.097328 31302 raft_consensus.cc:374] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.097338 31300 raft_consensus.cc:374] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.097404 31300 raft_consensus.cc:729] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.097412 31302 raft_consensus.cc:729] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.097412 31306 raft_consensus.cc:729] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.097627 31300 consensus_queue.cc:260] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.097669 31302 consensus_queue.cc:260] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.097668 31306 consensus_queue.cc:260] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.098368 31306 ts_tablet_manager.cc:1398] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:29.099085 31300 ts_tablet_manager.cc:1398] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:29.099543 31302 ts_tablet_manager.cc:1398] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:29.099777 31302 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:29.100488 31300 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:29.101328 31302 tablet_bootstrap.cc:654] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.103040 31300 tablet_bootstrap.cc:654] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.105137 31300 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:29.105391 31300 ts_tablet_manager.cc:1367] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:29.106523 31300 raft_consensus.cc:348] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.106767 31300 raft_consensus.cc:374] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.106850 31300 raft_consensus.cc:729] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.107122 31300 consensus_queue.cc:260] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.107767 31300 ts_tablet_manager.cc:1398] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:29.108081 31300 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:29.109625 31300 tablet_bootstrap.cc:654] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.109700 31302 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:29.109851 31302 ts_tablet_manager.cc:1367] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 14:00:29.110628 31302 raft_consensus.cc:348] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.110872 31302 raft_consensus.cc:374] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.111106 31302 raft_consensus.cc:729] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.111377 31302 consensus_queue.cc:260] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.111879 31302 ts_tablet_manager.cc:1398] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:29.114154 31300 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:29.114326 31300 ts_tablet_manager.cc:1367] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:29.114923 31300 raft_consensus.cc:348] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.115953 31300 raft_consensus.cc:374] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.116114 31300 raft_consensus.cc:729] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.116392 31300 consensus_queue.cc:260] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.116735 31305 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:29.116884 31300 ts_tablet_manager.cc:1398] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:29.116945 31305 ts_tablet_manager.cc:1367] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20240501 14:00:29.117604 31305 raft_consensus.cc:348] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.117862 31305 raft_consensus.cc:374] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.117971 31305 raft_consensus.cc:729] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.118224 31305 consensus_queue.cc:260] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.118889 31305 ts_tablet_manager.cc:1398] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:29.119235 31305 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:29.120584 31305 tablet_bootstrap.cc:654] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.122630 31305 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:29.122790 31305 ts_tablet_manager.cc:1367] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:00:29.123437 31305 raft_consensus.cc:348] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.123661 31305 raft_consensus.cc:374] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.123736 31305 raft_consensus.cc:729] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.123939 31305 consensus_queue.cc:260] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.124408 31305 ts_tablet_manager.cc:1398] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:29.124835 31305 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:29.126034 31305 tablet_bootstrap.cc:654] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:29.127679 31305 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:29.127805 31305 ts_tablet_manager.cc:1367] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:29.128365 31305 raft_consensus.cc:348] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.128513 31305 raft_consensus.cc:374] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:29.128644 31305 raft_consensus.cc:729] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:29.128832 31305 consensus_queue.cc:260] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.129359 31305 ts_tablet_manager.cc:1398] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:00:29.155309 31179 raft_consensus.cc:482] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:29.155493 31179 raft_consensus.cc:504] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.156239 31179 leader_election.cc:288] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:29.156625 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c63489706e4d068adde1872f451ad2" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:29.156685 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c63489706e4d068adde1872f451ad2" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:29.156837 29505 raft_consensus.cc:2454] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:29.156911 29246 raft_consensus.cc:2454] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:29.157272 29311 leader_election.cc:302] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:29.157555 31179 raft_consensus.cc:2781] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:29.157666 31179 raft_consensus.cc:482] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:29.157759 31179 raft_consensus.cc:3037] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.159811 31179 raft_consensus.cc:504] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.160359 31179 leader_election.cc:288] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:29.160902 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c63489706e4d068adde1872f451ad2" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:29.160933 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c63489706e4d068adde1872f451ad2" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:29.161137 29505 raft_consensus.cc:3037] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.161144 29246 raft_consensus.cc:3037] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.163038 29246 raft_consensus.cc:2454] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:29.163046 29505 raft_consensus.cc:2454] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:29.163452 29311 leader_election.cc:302] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:29.163770 31179 raft_consensus.cc:2781] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:29.163935 31179 raft_consensus.cc:686] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:29.164235 31179 consensus_queue.cc:237] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:29.167604 29103 catalog_manager.cc:5484] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:29.265347 31040 raft_consensus.cc:482] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:29.265544 31040 raft_consensus.cc:504] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.266204 31040 leader_election.cc:288] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:29.266697 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192a6cc308da4a448f03717b6a439094" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:29.266721 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192a6cc308da4a448f03717b6a439094" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:29.266917 29375 raft_consensus.cc:2454] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:29.266933 29505 raft_consensus.cc:2454] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:29.267360 29572 leader_election.cc:302] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:29.267611 31040 raft_consensus.cc:2781] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:29.267728 31040 raft_consensus.cc:482] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:29.267819 31040 raft_consensus.cc:3037] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.269729 31040 raft_consensus.cc:504] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.270299 31040 leader_election.cc:288] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:29.270704 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192a6cc308da4a448f03717b6a439094" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:29.270735 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192a6cc308da4a448f03717b6a439094" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:29.270879 29375 raft_consensus.cc:3037] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.270906 29505 raft_consensus.cc:3037] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.272676 29505 raft_consensus.cc:2454] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:29.272753 29375 raft_consensus.cc:2454] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:29.273082 29571 leader_election.cc:302] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:29.273365 31040 raft_consensus.cc:2781] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:29.273718 31040 raft_consensus.cc:686] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:29.274047 31040 consensus_queue.cc:237] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.277135 31040 raft_consensus.cc:482] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:29.277035 29103 catalog_manager.cc:5484] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:29.277345 31040 raft_consensus.cc:504] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.278038 31040 leader_election.cc:288] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:29.278362 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e513634d27c5467fb49c9717c43904cd" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:29.278581 29375 raft_consensus.cc:2454] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:29.278544 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e513634d27c5467fb49c9717c43904cd" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:29.278735 29505 raft_consensus.cc:2454] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:29.279078 29572 leader_election.cc:302] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:29.279525 31040 raft_consensus.cc:2781] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:29.279601 31040 raft_consensus.cc:482] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:29.279697 31040 raft_consensus.cc:3037] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.281785 31040 raft_consensus.cc:504] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.282328 31040 leader_election.cc:288] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:29.282737 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e513634d27c5467fb49c9717c43904cd" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:29.282799 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e513634d27c5467fb49c9717c43904cd" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:29.282969 29505 raft_consensus.cc:3037] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.282969 29375 raft_consensus.cc:3037] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.285787 29505 raft_consensus.cc:2454] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:29.285774 29375 raft_consensus.cc:2454] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:29.286164 29571 leader_election.cc:302] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:29.286423 31040 raft_consensus.cc:2781] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:29.286589 31040 raft_consensus.cc:686] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:29.286917 31040 consensus_queue.cc:237] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:29.290627 29103 catalog_manager.cc:5484] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:29.662389 31179 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.671977 31215 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.687568 31040 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.694515 31326 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:29.713078 31318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:29.713418 31318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:29.721133 31318 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 14:00:29.782115 31326 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.800827 31315 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 297.392 ms
time per row: 0.148696 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:31.135649 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:31.144730 29226 tablet_service.cc:1467] Processing CreateTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.144779 29616 tablet_service.cc:1467] Processing CreateTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.145274 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670a438f0a94bc0814536c5a8ecbf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.145272 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670a438f0a94bc0814536c5a8ecbf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.145704 29617 tablet_service.cc:1467] Processing CreateTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.145996 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b679ed4f63414ae1aefe8cd769b4b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.149732 29225 tablet_service.cc:1467] Processing CreateTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.149814 29357 tablet_service.cc:1467] Processing CreateTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.149871 29615 tablet_service.cc:1467] Processing CreateTablet for tablet db7077d63a534acc94c56c2b6e7b1856 (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.150130 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b679ed4f63414ae1aefe8cd769b4b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.150182 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e670a438f0a94bc0814536c5a8ecbf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.150226 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7077d63a534acc94c56c2b6e7b1856. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.152806 29224 tablet_service.cc:1467] Processing CreateTablet for tablet db7077d63a534acc94c56c2b6e7b1856 (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.153177 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7077d63a534acc94c56c2b6e7b1856. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.153717 29355 tablet_service.cc:1467] Processing CreateTablet for tablet db7077d63a534acc94c56c2b6e7b1856 (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.153718 29487 tablet_service.cc:1467] Processing CreateTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:31.154064 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7077d63a534acc94c56c2b6e7b1856. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.154138 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b679ed4f63414ae1aefe8cd769b4b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.174448 31389 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:31.176075 31387 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:31.176352 31388 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:31.177775 31389 tablet_bootstrap.cc:654] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.177842 31387 tablet_bootstrap.cc:654] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.177964 31388 tablet_bootstrap.cc:654] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.179519 31386 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:31.181612 31386 tablet_bootstrap.cc:654] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.182185 31389 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:31.182281 31388 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:31.182412 31389 ts_tablet_manager.cc:1367] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:00:31.182433 31388 ts_tablet_manager.cc:1367] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:31.183187 31389 raft_consensus.cc:348] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.183187 31388 raft_consensus.cc:348] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.183389 31388 raft_consensus.cc:374] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.183499 31388 raft_consensus.cc:729] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.183509 31389 raft_consensus.cc:374] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.183653 31389 raft_consensus.cc:729] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.183773 31388 consensus_queue.cc:260] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.184459 31388 ts_tablet_manager.cc:1398] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:31.184764 31388 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:31.183904 31389 consensus_queue.cc:260] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.185547 31389 ts_tablet_manager.cc:1398] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:31.186543 31388 tablet_bootstrap.cc:654] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.187947 31386 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:31.188161 31386 ts_tablet_manager.cc:1367] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240501 14:00:31.189021 31386 raft_consensus.cc:348] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.189340 31386 raft_consensus.cc:374] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.189504 31386 raft_consensus.cc:729] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.189790 31386 consensus_queue.cc:260] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.190554 31386 ts_tablet_manager.cc:1398] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:31.190899 31386 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:31.192613 31386 tablet_bootstrap.cc:654] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.197100 31387 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:31.197332 31387 ts_tablet_manager.cc:1367] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20240501 14:00:31.198756 31387 raft_consensus.cc:348] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.199075 31387 raft_consensus.cc:374] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.199191 31387 raft_consensus.cc:729] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.199523 31387 consensus_queue.cc:260] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.200271 31387 ts_tablet_manager.cc:1398] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:31.200600 31387 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:31.201157 31386 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:31.201388 31386 ts_tablet_manager.cc:1367] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 14:00:31.202227 31386 raft_consensus.cc:348] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.202570 31386 raft_consensus.cc:374] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.202697 31386 raft_consensus.cc:729] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.203023 31386 consensus_queue.cc:260] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.203605 31386 ts_tablet_manager.cc:1398] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:31.204028 31386 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:31.204254 31387 tablet_bootstrap.cc:654] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.205653 31386 tablet_bootstrap.cc:654] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.211836 31388 tablet_bootstrap.cc:492] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:31.212152 31388 ts_tablet_manager.cc:1367] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.012s
I20240501 14:00:31.213348 31388 raft_consensus.cc:348] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.213588 31386 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:31.213675 31388 raft_consensus.cc:374] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.213726 31386 ts_tablet_manager.cc:1367] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:00:31.213773 31388 raft_consensus.cc:729] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.214089 31388 consensus_queue.cc:260] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.214371 31386 raft_consensus.cc:348] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.214597 31386 raft_consensus.cc:374] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.214702 31386 raft_consensus.cc:729] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.214917 31386 consensus_queue.cc:260] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.216195 31388 ts_tablet_manager.cc:1398] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:31.216637 31388 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:31.216707 31386 ts_tablet_manager.cc:1398] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:31.218885 31388 tablet_bootstrap.cc:654] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:31.224944 31315 raft_consensus.cc:482] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.225155 31315 raft_consensus.cc:504] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.226133 31315 leader_election.cc:288] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:31.226505 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7077d63a534acc94c56c2b6e7b1856" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:31.227191 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7077d63a534acc94c56c2b6e7b1856" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:31.227406 29246 raft_consensus.cc:2454] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
W20240501 14:00:31.227389 29572 leader_election.cc:341] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:31.227819 29570 leader_election.cc:302] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:00:31.228226 31315 raft_consensus.cc:2781] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.228318 31315 raft_consensus.cc:482] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.228420 31315 raft_consensus.cc:3037] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.228948 31387 tablet_bootstrap.cc:492] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:31.229195 31387 ts_tablet_manager.cc:1367] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.014s
I20240501 14:00:31.230203 31387 raft_consensus.cc:348] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.230520 31387 raft_consensus.cc:374] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.230469 31315 raft_consensus.cc:504] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.230628 31387 raft_consensus.cc:729] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.230942 31387 consensus_queue.cc:260] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.231206 31315 leader_election.cc:288] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:31.231740 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7077d63a534acc94c56c2b6e7b1856" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:31.231747 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7077d63a534acc94c56c2b6e7b1856" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:31.231935 29246 raft_consensus.cc:3037] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.231948 29376 raft_consensus.cc:3037] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.233203 31387 ts_tablet_manager.cc:1398] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:31.234555 31388 tablet_bootstrap.cc:492] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:31.234731 31388 ts_tablet_manager.cc:1367] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 14:00:31.235491 29376 raft_consensus.cc:2454] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:31.235555 31388 raft_consensus.cc:348] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.235767 31388 raft_consensus.cc:374] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:31.235874 31388 raft_consensus.cc:729] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.236019 29572 leader_election.cc:302] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:31.236187 31388 consensus_queue.cc:260] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.236752 31388 ts_tablet_manager.cc:1398] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:31.237308 31315 raft_consensus.cc:2781] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.238750 29246 raft_consensus.cc:2454] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:31.239801 31315 raft_consensus.cc:686] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:31.240120 31315 consensus_queue.cc:237] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:31.243100 29100 catalog_manager.cc:5484] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:31.280349 31215 raft_consensus.cc:482] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.280584 31215 raft_consensus.cc:504] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.281772 31215 leader_election.cc:288] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:31.282414 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670a438f0a94bc0814536c5a8ecbf9e" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:31.282563 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670a438f0a94bc0814536c5a8ecbf9e" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:31.282688 29246 raft_consensus.cc:2454] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:31.282763 29636 raft_consensus.cc:2454] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:31.283201 29310 leader_election.cc:302] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:31.283509 31215 raft_consensus.cc:2781] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.283651 31215 raft_consensus.cc:482] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.283758 31215 raft_consensus.cc:3037] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.285882 31215 raft_consensus.cc:504] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.286402 31215 leader_election.cc:288] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:31.286868 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670a438f0a94bc0814536c5a8ecbf9e" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:31.286939 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e670a438f0a94bc0814536c5a8ecbf9e" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:31.287091 29246 raft_consensus.cc:3037] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.287128 29636 raft_consensus.cc:3037] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.289381 29246 raft_consensus.cc:2454] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:31.289572 29636 raft_consensus.cc:2454] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:31.289795 29310 leader_election.cc:302] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:31.290061 31215 raft_consensus.cc:2781] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.290222 31215 raft_consensus.cc:686] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:31.290488 31215 consensus_queue.cc:237] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:31.293596 29100 catalog_manager.cc:5484] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:31.513947 31315 raft_consensus.cc:482] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.514144 31315 raft_consensus.cc:504] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.514907 31315 leader_election.cc:288] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:31.515411 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b679ed4f63414ae1aefe8cd769b4b05c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:31.515450 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b679ed4f63414ae1aefe8cd769b4b05c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:31.515643 29505 raft_consensus.cc:2454] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:31.515662 29246 raft_consensus.cc:2454] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:31.516012 29570 leader_election.cc:302] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:31.516281 31315 raft_consensus.cc:2781] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.516379 31315 raft_consensus.cc:482] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.516471 31315 raft_consensus.cc:3037] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.518381 31315 raft_consensus.cc:504] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.518915 31315 leader_election.cc:288] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:31.519345 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b679ed4f63414ae1aefe8cd769b4b05c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:31.519455 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b679ed4f63414ae1aefe8cd769b4b05c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:31.519565 29505 raft_consensus.cc:3037] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.519644 29246 raft_consensus.cc:3037] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.521514 29505 raft_consensus.cc:2454] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:31.521541 29246 raft_consensus.cc:2454] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:31.521888 29571 leader_election.cc:302] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:31.522167 31315 raft_consensus.cc:2781] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.522326 31315 raft_consensus.cc:686] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:31.522596 31315 consensus_queue.cc:237] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:31.525858 29100 catalog_manager.cc:5484] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:31.693190 31215 consensus_queue.cc:1035] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.702481 31381 consensus_queue.cc:1035] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.776268 31315 consensus_queue.cc:1035] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.782368 31315 consensus_queue.cc:1035] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:31.902026 31403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:31.902407 31403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:31.910240 31403 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 14:00:31.955966 31410 consensus_queue.cc:1035] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.961460 31410 consensus_queue.cc:1035] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.994 ms
time per row: 0.136497 ms
I20240501 14:00:32.341094 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:32.350411 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 71933eccec5f4fdc8b1df7111f941777 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.350977 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71933eccec5f4fdc8b1df7111f941777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.351123 29355 tablet_service.cc:1467] Processing CreateTablet for tablet a82cbc0696fe4a8a815eaf0538b5aad8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.351545 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82cbc0696fe4a8a815eaf0538b5aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.351542 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 71933eccec5f4fdc8b1df7111f941777 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.351943 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71933eccec5f4fdc8b1df7111f941777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.353432 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 71933eccec5f4fdc8b1df7111f941777 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.353955 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71933eccec5f4fdc8b1df7111f941777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.354555 29486 tablet_service.cc:1467] Processing CreateTablet for tablet a82cbc0696fe4a8a815eaf0538b5aad8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.354936 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82cbc0696fe4a8a815eaf0538b5aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.357398 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 8b925d954c9143279ba69ae5c90d3266 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.357789 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b925d954c9143279ba69ae5c90d3266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.358577 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 8b925d954c9143279ba69ae5c90d3266 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.359110 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b925d954c9143279ba69ae5c90d3266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.361390 29617 tablet_service.cc:1467] Processing CreateTablet for tablet a82cbc0696fe4a8a815eaf0538b5aad8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.361832 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82cbc0696fe4a8a815eaf0538b5aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.365388 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 8b925d954c9143279ba69ae5c90d3266 (DEFAULT_TABLE table=rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:32.365813 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b925d954c9143279ba69ae5c90d3266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.380950 31467 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:32.381131 31468 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:32.382333 31469 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:32.383353 31467 tablet_bootstrap.cc:654] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.384322 31468 tablet_bootstrap.cc:654] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.385337 31467 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:32.385605 31467 ts_tablet_manager.cc:1367] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:32.385645 31469 tablet_bootstrap.cc:654] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.386437 31467 raft_consensus.cc:348] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.386612 31467 raft_consensus.cc:374] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.386721 31467 raft_consensus.cc:729] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.386955 31467 consensus_queue.cc:260] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.387519 31466 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:32.387712 31467 ts_tablet_manager.cc:1398] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:32.388190 31467 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:32.389544 31466 tablet_bootstrap.cc:654] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.389586 31468 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:32.389768 31468 ts_tablet_manager.cc:1367] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:00:32.390781 31468 raft_consensus.cc:348] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.391649 31469 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:32.391866 31469 ts_tablet_manager.cc:1367] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 14:00:32.391091 31468 raft_consensus.cc:374] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.392205 31468 raft_consensus.cc:729] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.392454 31468 consensus_queue.cc:260] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.392774 31469 raft_consensus.cc:348] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.393078 31469 raft_consensus.cc:374] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.393219 31468 ts_tablet_manager.cc:1398] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:32.393285 31469 raft_consensus.cc:729] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.393663 31468 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:32.393587 31469 consensus_queue.cc:260] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.394351 31469 ts_tablet_manager.cc:1398] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:32.394753 31469 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:32.395524 31468 tablet_bootstrap.cc:654] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.395984 31467 tablet_bootstrap.cc:654] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.396478 31469 tablet_bootstrap.cc:654] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.399701 31466 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:32.399883 31466 ts_tablet_manager.cc:1367] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:00:32.400735 31466 raft_consensus.cc:348] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.400964 31466 raft_consensus.cc:374] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.401103 31466 raft_consensus.cc:729] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.401352 31466 consensus_queue.cc:260] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.402072 31466 ts_tablet_manager.cc:1398] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:32.402374 31466 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:32.404265 31466 tablet_bootstrap.cc:654] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.419838 31467 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:32.423336 31469 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:32.423673 31469 ts_tablet_manager.cc:1367] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20240501 14:00:32.424723 31469 raft_consensus.cc:348] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.425127 31469 raft_consensus.cc:374] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.425279 31469 raft_consensus.cc:729] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.425693 31469 consensus_queue.cc:260] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.426501 31469 ts_tablet_manager.cc:1398] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:32.434501 31468 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:32.434806 31468 ts_tablet_manager.cc:1367] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.012s
I20240501 14:00:32.435835 31410 raft_consensus.cc:482] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:32.435745 31468 raft_consensus.cc:348] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.436012 31468 raft_consensus.cc:374] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.436025 31466 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:32.436082 31468 raft_consensus.cc:729] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.435998 31410 raft_consensus.cc:504] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.436203 31466 ts_tablet_manager.cc:1367] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.011s
I20240501 14:00:32.436509 31468 consensus_queue.cc:260] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.436812 31410 leader_election.cc:288] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:32.437130 31468 ts_tablet_manager.cc:1398] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:32.437979 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:32.437181 31466 raft_consensus.cc:348] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.438351 31466 raft_consensus.cc:374] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.438479 31466 raft_consensus.cc:729] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.438827 31466 consensus_queue.cc:260] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.439154 29375 raft_consensus.cc:2454] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:32.440007 29572 leader_election.cc:302] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:32.440389 31410 raft_consensus.cc:2781] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:32.440505 31410 raft_consensus.cc:482] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:32.440609 31410 raft_consensus.cc:3037] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.441627 31466 ts_tablet_manager.cc:1398] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:32.441612 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
W20240501 14:00:32.443104 29571 leader_election.cc:341] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:32.445127 31410 raft_consensus.cc:504] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.420323 31467 ts_tablet_manager.cc:1367] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.000s
I20240501 14:00:32.446770 31410 leader_election.cc:288] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:32.447005 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:32.447059 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:32.447273 29376 raft_consensus.cc:3037] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:32.447831 29571 leader_election.cc:341] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:32.448221 31467 raft_consensus.cc:348] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.448521 31467 raft_consensus.cc:374] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.448662 31467 raft_consensus.cc:729] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.449064 31467 consensus_queue.cc:260] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.449731 31467 ts_tablet_manager.cc:1398] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:32.450073 31467 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:32.450101 29376 raft_consensus.cc:2454] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:32.451777 31467 tablet_bootstrap.cc:654] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:32.454406 29572 leader_election.cc:302] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:00:32.455072 31410 raft_consensus.cc:2781] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:32.455912 31410 raft_consensus.cc:686] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:32.456323 31410 consensus_queue.cc:237] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.461503 31467 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:32.461748 31467 ts_tablet_manager.cc:1367] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:00:32.462697 31467 raft_consensus.cc:348] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.462999 31467 raft_consensus.cc:374] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:32.463156 31467 raft_consensus.cc:729] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.463476 31467 consensus_queue.cc:260] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.464192 31467 ts_tablet_manager.cc:1398] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:32.491387 29098 catalog_manager.cc:5484] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:32.631172 31244 raft_consensus.cc:482] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:32.631464 31244 raft_consensus.cc:504] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.632194 31244 leader_election.cc:288] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:32.632620 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b925d954c9143279ba69ae5c90d3266" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:32.632846 29505 raft_consensus.cc:2454] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:32.632922 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b925d954c9143279ba69ae5c90d3266" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:32.633239 29637 raft_consensus.cc:2454] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:32.633378 29180 leader_election.cc:302] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:32.633630 31244 raft_consensus.cc:2781] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:32.633795 31244 raft_consensus.cc:482] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:32.633883 31244 raft_consensus.cc:3037] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.635854 31244 raft_consensus.cc:504] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.636433 31244 leader_election.cc:288] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:32.636852 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b925d954c9143279ba69ae5c90d3266" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:32.637055 29505 raft_consensus.cc:3037] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.636952 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b925d954c9143279ba69ae5c90d3266" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:32.637231 29637 raft_consensus.cc:3037] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.639151 29505 raft_consensus.cc:2454] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:32.639173 29637 raft_consensus.cc:2454] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:32.639580 29180 leader_election.cc:302] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:32.639858 31244 raft_consensus.cc:2781] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:32.640000 31244 raft_consensus.cc:686] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:32.640360 31244 consensus_queue.cc:237] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.643997 29098 catalog_manager.cc:5484] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:32.659679 31456 raft_consensus.cc:482] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:32.659895 31456 raft_consensus.cc:504] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.660895 31456 leader_election.cc:288] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:32.661391 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71933eccec5f4fdc8b1df7111f941777" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:32.661518 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71933eccec5f4fdc8b1df7111f941777" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:32.661666 29505 raft_consensus.cc:2454] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:32.661770 29244 raft_consensus.cc:2454] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:32.662206 29311 leader_election.cc:302] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:32.662477 31456 raft_consensus.cc:2781] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:32.662604 31456 raft_consensus.cc:482] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:32.662711 31456 raft_consensus.cc:3037] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.664726 31456 raft_consensus.cc:504] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.665342 31456 leader_election.cc:288] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:32.665908 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71933eccec5f4fdc8b1df7111f941777" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:32.665885 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71933eccec5f4fdc8b1df7111f941777" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:32.666114 29244 raft_consensus.cc:3037] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.666137 29505 raft_consensus.cc:3037] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.669140 29244 raft_consensus.cc:2454] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:32.669134 29505 raft_consensus.cc:2454] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:32.669601 29310 leader_election.cc:302] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:32.669961 31456 raft_consensus.cc:2781] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:32.670146 31456 raft_consensus.cc:686] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:32.670456 31456 consensus_queue.cc:237] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:32.672853 31261 raft_consensus.cc:482] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:32.673041 31261 raft_consensus.cc:504] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:32.674044 31261 leader_election.cc:288] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:32.674105 29098 catalog_manager.cc:5484] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:32.674419 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:32.674695 29376 raft_consensus.cc:2379] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 522f4ff5e9084cbf8462196533c7d7aa in current term 1: Already voted for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in this term.
I20240501 14:00:32.675001 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82cbc0696fe4a8a815eaf0538b5aad8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:32.675594 29440 leader_election.cc:302] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:32.675863 31261 raft_consensus.cc:3037] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:32.677982 31261 raft_consensus.cc:2726] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:32.934265 31461 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:32.958643 31410 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:33.075488 31381 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:33.080917 31456 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:33.087673 31482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:33.088155 31482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:33.097005 31482 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 14:00:33.154812 31244 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:33.160413 31232 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 266.424 ms
time per row: 0.133212 ms
I20240501 14:00:33.551352 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:33.560312 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.560871 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec4a91f29449f48ac3fcf10d13b6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.561316 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 040227e018da4e3db04c371c4510b82a (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.561425 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 040227e018da4e3db04c371c4510b82a (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.561717 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040227e018da4e3db04c371c4510b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.561811 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040227e018da4e3db04c371c4510b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.562731 29356 tablet_service.cc:1467] Processing CreateTablet for tablet a61f994801b643e1a369a034580cf3ed (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.563160 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f994801b643e1a369a034580cf3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.565824 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 040227e018da4e3db04c371c4510b82a (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.566138 29617 tablet_service.cc:1467] Processing CreateTablet for tablet a61f994801b643e1a369a034580cf3ed (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.566268 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040227e018da4e3db04c371c4510b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.566537 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f994801b643e1a369a034580cf3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.569198 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.569677 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec4a91f29449f48ac3fcf10d13b6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.573148 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.573740 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec4a91f29449f48ac3fcf10d13b6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.574467 29486 tablet_service.cc:1467] Processing CreateTablet for tablet a61f994801b643e1a369a034580cf3ed (DEFAULT_TABLE table=rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:33.574894 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f994801b643e1a369a034580cf3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.587152 31549 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:33.587211 31548 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:33.588629 31549 tablet_bootstrap.cc:654] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.588644 31548 tablet_bootstrap.cc:654] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.590297 31550 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:33.591055 31548 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:33.591209 31548 ts_tablet_manager.cc:1367] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:33.591913 31550 tablet_bootstrap.cc:654] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.591971 31548 raft_consensus.cc:348] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.592195 31548 raft_consensus.cc:374] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.592291 31548 raft_consensus.cc:729] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.592525 31548 consensus_queue.cc:260] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.593319 31548 ts_tablet_manager.cc:1398] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:33.594650 31549 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:33.594882 31549 ts_tablet_manager.cc:1367] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240501 14:00:33.595400 31548 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:33.595732 31549 raft_consensus.cc:348] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.596174 31549 raft_consensus.cc:374] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.596330 31549 raft_consensus.cc:729] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.596657 31549 consensus_queue.cc:260] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.597358 31549 ts_tablet_manager.cc:1398] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:33.597968 31548 tablet_bootstrap.cc:654] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.598183 31550 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:33.598390 31550 ts_tablet_manager.cc:1367] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:00:33.598721 31549 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:33.599606 31550 raft_consensus.cc:348] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.599956 31550 raft_consensus.cc:374] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.600107 31550 raft_consensus.cc:729] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.600405 31550 consensus_queue.cc:260] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.601892 31550 ts_tablet_manager.cc:1398] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:33.602444 31549 tablet_bootstrap.cc:654] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.603663 31553 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:33.607159 31553 tablet_bootstrap.cc:654] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.608328 31550 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:33.610307 31550 tablet_bootstrap.cc:654] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.610723 31548 tablet_bootstrap.cc:492] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:33.611016 31548 ts_tablet_manager.cc:1367] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20240501 14:00:33.611917 31548 raft_consensus.cc:348] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.612294 31548 raft_consensus.cc:374] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.612424 31548 raft_consensus.cc:729] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.612807 31548 consensus_queue.cc:260] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.614102 31549 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:33.614377 31549 ts_tablet_manager.cc:1367] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20240501 14:00:33.615146 31548 ts_tablet_manager.cc:1398] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:33.615603 31548 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:33.615588 31549 raft_consensus.cc:348] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.615891 31549 raft_consensus.cc:374] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.616108 31549 raft_consensus.cc:729] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.616446 31549 consensus_queue.cc:260] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.618309 31549 ts_tablet_manager.cc:1398] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:33.619279 31548 tablet_bootstrap.cc:654] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.619936 31553 tablet_bootstrap.cc:492] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:33.620154 31553 ts_tablet_manager.cc:1367] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240501 14:00:33.621441 31553 raft_consensus.cc:348] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.621649 31550 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:33.621731 31553 raft_consensus.cc:374] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.621943 31550 ts_tablet_manager.cc:1367] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20240501 14:00:33.623032 31550 raft_consensus.cc:348] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.623432 31550 raft_consensus.cc:374] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.623579 31550 raft_consensus.cc:729] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.623796 31553 raft_consensus.cc:729] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.623953 31550 consensus_queue.cc:260] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.624106 31553 consensus_queue.cc:260] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.624851 31550 ts_tablet_manager.cc:1398] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:33.626792 31553 ts_tablet_manager.cc:1398] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:33.627177 31553 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:33.631220 31553 tablet_bootstrap.cc:654] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.635290 31548 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:33.635471 31548 ts_tablet_manager.cc:1367] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.010s
I20240501 14:00:33.635874 31553 tablet_bootstrap.cc:492] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:33.636044 31553 ts_tablet_manager.cc:1367] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240501 14:00:33.636359 31548 raft_consensus.cc:348] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.636590 31548 raft_consensus.cc:374] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.636703 31548 raft_consensus.cc:729] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.636880 31553 raft_consensus.cc:348] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.637117 31553 raft_consensus.cc:374] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.636945 31548 consensus_queue.cc:260] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.637193 31553 raft_consensus.cc:729] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.637678 31548 ts_tablet_manager.cc:1398] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:33.637727 31553 consensus_queue.cc:260] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.639320 31553 ts_tablet_manager.cc:1398] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:33.690132 31456 raft_consensus.cc:482] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:33.690312 31456 raft_consensus.cc:504] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.690943 31456 leader_election.cc:288] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:33.691373 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec4a91f29449f48ac3fcf10d13b6d4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:33.691453 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec4a91f29449f48ac3fcf10d13b6d4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:33.691646 29506 raft_consensus.cc:2454] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:33.691711 29245 raft_consensus.cc:2454] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:33.692174 29310 leader_election.cc:302] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:33.692463 31456 raft_consensus.cc:2781] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:33.692575 31456 raft_consensus.cc:482] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:33.692667 31456 raft_consensus.cc:3037] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.694679 31456 raft_consensus.cc:504] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.695266 31456 leader_election.cc:288] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:33.695655 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec4a91f29449f48ac3fcf10d13b6d4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:33.695725 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec4a91f29449f48ac3fcf10d13b6d4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:33.695899 29506 raft_consensus.cc:3037] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.695973 29245 raft_consensus.cc:3037] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.698928 29506 raft_consensus.cc:2454] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:33.698923 29245 raft_consensus.cc:2454] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:33.699410 29310 leader_election.cc:302] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:33.699725 31456 raft_consensus.cc:2781] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:33.699892 31456 raft_consensus.cc:686] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:33.700204 31456 consensus_queue.cc:237] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.702154 31542 raft_consensus.cc:482] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:33.702314 31542 raft_consensus.cc:504] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.703030 31542 leader_election.cc:288] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:33.703501 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f994801b643e1a369a034580cf3ed" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:33.703526 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f994801b643e1a369a034580cf3ed" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:33.703718 29506 raft_consensus.cc:2454] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:33.703718 29375 raft_consensus.cc:2454] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:33.703747 29097 catalog_manager.cc:5484] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:33.704169 29571 leader_election.cc:302] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:33.704483 31542 raft_consensus.cc:2781] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:33.704620 31542 raft_consensus.cc:482] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:33.704720 31542 raft_consensus.cc:3037] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.706939 31542 raft_consensus.cc:504] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.707541 31542 leader_election.cc:288] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:33.707965 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f994801b643e1a369a034580cf3ed" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:33.708077 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f994801b643e1a369a034580cf3ed" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:33.708163 29375 raft_consensus.cc:3037] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.708284 29506 raft_consensus.cc:3037] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.710883 29506 raft_consensus.cc:2454] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:33.711014 29375 raft_consensus.cc:2454] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:33.711480 29571 leader_election.cc:302] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:33.711911 31542 raft_consensus.cc:2781] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:33.712059 31542 raft_consensus.cc:686] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:33.712366 31542 consensus_queue.cc:237] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:33.715678 29097 catalog_manager.cc:5484] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:33.752041 31542 raft_consensus.cc:482] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:33.752306 31542 raft_consensus.cc:504] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.753082 31542 leader_election.cc:288] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:33.753671 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040227e018da4e3db04c371c4510b82a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:33.753929 29375 raft_consensus.cc:2454] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:33.754022 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040227e018da4e3db04c371c4510b82a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:33.754283 29245 raft_consensus.cc:2454] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:33.754333 29572 leader_election.cc:302] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:33.754673 31542 raft_consensus.cc:2781] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:33.754778 31542 raft_consensus.cc:482] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:33.754886 31542 raft_consensus.cc:3037] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.756868 31542 raft_consensus.cc:504] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.757926 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040227e018da4e3db04c371c4510b82a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:33.758008 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040227e018da4e3db04c371c4510b82a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:33.758134 31542 leader_election.cc:288] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:33.758211 29245 raft_consensus.cc:3037] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.758328 29375 raft_consensus.cc:3037] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.761188 29245 raft_consensus.cc:2454] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:33.761181 29375 raft_consensus.cc:2454] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:33.761689 29572 leader_election.cc:302] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:33.762027 31542 raft_consensus.cc:2781] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:33.762176 31542 raft_consensus.cc:686] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:33.762446 31542 consensus_queue.cc:237] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:33.765799 29097 catalog_manager.cc:5484] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
W20240501 14:00:34.145608 31564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:34.146036 31564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:00:34.147291 31542 consensus_queue.cc:1035] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:34.158576 31564 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:34.163322 31461 consensus_queue.cc:1035] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240501 14:00:34.238413 31461 consensus_queue.cc:1035] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:34.245883 31542 consensus_queue.cc:1035] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:34.285251 31529 consensus_queue.cc:1035] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:34.289181 31456 consensus_queue.cc:1035] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 336.382 ms
time per row: 0.168191 ms
I20240501 14:00:34.698488 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:34.707523 29224 tablet_service.cc:1467] Processing CreateTablet for tablet ba20c01d741a414196972ced1601baf5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.708012 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20c01d741a414196972ced1601baf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.708005 29616 tablet_service.cc:1467] Processing CreateTablet for tablet ba20c01d741a414196972ced1601baf5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.708436 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20c01d741a414196972ced1601baf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.708374 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 60527f81f34b45c199bf52bf0df53ee2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.708761 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60527f81f34b45c199bf52bf0df53ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.709112 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 60527f81f34b45c199bf52bf0df53ee2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.709372 29485 tablet_service.cc:1467] Processing CreateTablet for tablet ba20c01d741a414196972ced1601baf5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.709491 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60527f81f34b45c199bf52bf0df53ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.709775 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20c01d741a414196972ced1601baf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.710939 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8249c87f42faa9a7464369048090 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.711342 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8249c87f42faa9a7464369048090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.712724 29486 tablet_service.cc:1467] Processing CreateTablet for tablet 60527f81f34b45c199bf52bf0df53ee2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.713066 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60527f81f34b45c199bf52bf0df53ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.713433 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8249c87f42faa9a7464369048090 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.713761 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8249c87f42faa9a7464369048090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.722560 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8249c87f42faa9a7464369048090 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.722983 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8249c87f42faa9a7464369048090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.723511 31637 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:34.728160 31637 tablet_bootstrap.cc:654] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.730911 31634 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:34.735316 31635 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:34.736037 31634 tablet_bootstrap.cc:654] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.737304 31636 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:34.737673 31635 tablet_bootstrap.cc:654] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.739850 31636 tablet_bootstrap.cc:654] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.741729 31634 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:34.741879 31634 ts_tablet_manager.cc:1367] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 14:00:34.744282 31634 raft_consensus.cc:348] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.744479 31634 raft_consensus.cc:374] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.744585 31634 raft_consensus.cc:729] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.745038 31634 consensus_queue.cc:260] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.745781 31634 ts_tablet_manager.cc:1398] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:34.746093 31634 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:34.746831 31636 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:34.747113 31636 ts_tablet_manager.cc:1367] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240501 14:00:34.748111 31636 raft_consensus.cc:348] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.748445 31636 raft_consensus.cc:374] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.748483 31634 tablet_bootstrap.cc:654] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.748622 31636 raft_consensus.cc:729] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.750324 31635 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:34.750512 31635 ts_tablet_manager.cc:1367] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 14:00:34.750660 31636 consensus_queue.cc:260] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.750844 31637 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:34.751333 31636 ts_tablet_manager.cc:1398] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:34.751525 31635 raft_consensus.cc:348] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.751765 31636 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:34.751783 31635 raft_consensus.cc:374] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.751886 31635 raft_consensus.cc:729] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.751969 31637 ts_tablet_manager.cc:1367] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.009s
I20240501 14:00:34.752156 31635 consensus_queue.cc:260] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.752921 31635 ts_tablet_manager.cc:1398] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:34.754106 31637 raft_consensus.cc:348] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.754292 31634 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:34.754439 31636 tablet_bootstrap.cc:654] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.754434 31637 raft_consensus.cc:374] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.754498 31634 ts_tablet_manager.cc:1367] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240501 14:00:34.754567 31637 raft_consensus.cc:729] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.754886 31637 consensus_queue.cc:260] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.755290 31634 raft_consensus.cc:348] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.755585 31634 raft_consensus.cc:374] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.755653 31637 ts_tablet_manager.cc:1398] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:34.755718 31634 raft_consensus.cc:729] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.756050 31634 consensus_queue.cc:260] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.756650 31634 ts_tablet_manager.cc:1398] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:34.759976 31635 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:34.763546 31635 tablet_bootstrap.cc:654] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.772640 31636 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:34.772640 31635 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:34.772837 31636 ts_tablet_manager.cc:1367] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20240501 14:00:34.772837 31635 ts_tablet_manager.cc:1367] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240501 14:00:34.773761 31635 raft_consensus.cc:348] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.773762 31636 raft_consensus.cc:348] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.774001 31636 raft_consensus.cc:374] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.774044 31635 raft_consensus.cc:374] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.774117 31636 raft_consensus.cc:729] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.774152 31635 raft_consensus.cc:729] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.774386 31635 consensus_queue.cc:260] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.774413 31636 consensus_queue.cc:260] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.774942 31635 ts_tablet_manager.cc:1398] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:34.775324 31636 ts_tablet_manager.cc:1398] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:34.775352 31635 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:34.775712 31636 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:34.777122 31635 tablet_bootstrap.cc:654] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.777402 31636 tablet_bootstrap.cc:654] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.788189 31635 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:34.788316 31636 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:34.788350 31635 ts_tablet_manager.cc:1367] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240501 14:00:34.788498 31636 ts_tablet_manager.cc:1367] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240501 14:00:34.789105 31635 raft_consensus.cc:348] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.789264 31635 raft_consensus.cc:374] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.789167 31636 raft_consensus.cc:348] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.789352 31636 raft_consensus.cc:374] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.789348 31635 raft_consensus.cc:729] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.789448 31636 raft_consensus.cc:729] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.789569 31635 consensus_queue.cc:260] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.789682 31636 consensus_queue.cc:260] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.790062 31635 ts_tablet_manager.cc:1398] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:34.790244 31636 ts_tablet_manager.cc:1398] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:34.793553 31244 raft_consensus.cc:482] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.793704 31244 raft_consensus.cc:504] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.794503 31244 leader_election.cc:288] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:34.795154 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:34.795369 29504 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:34.795688 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:34.795799 29180 leader_election.cc:302] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:34.795950 29637 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:34.796097 31244 raft_consensus.cc:2781] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.796214 31244 raft_consensus.cc:482] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.796308 31244 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.798801 31244 raft_consensus.cc:504] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.799531 31244 leader_election.cc:288] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:34.799958 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:34.800071 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:34.800148 29504 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.800310 29636 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.802265 29636 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:34.802389 29504 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:34.802668 29179 leader_election.cc:302] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:34.802959 31244 raft_consensus.cc:2781] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.803270 31244 raft_consensus.cc:686] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:34.803591 31244 consensus_queue.cc:237] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:34.806887 29096 catalog_manager.cc:5484] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:34.855662 31244 raft_consensus.cc:482] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.855852 31244 raft_consensus.cc:504] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.856473 31244 leader_election.cc:288] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:34.856933 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60527f81f34b45c199bf52bf0df53ee2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:34.856950 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60527f81f34b45c199bf52bf0df53ee2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:34.857193 29636 raft_consensus.cc:2454] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:34.857213 29504 raft_consensus.cc:2454] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:34.857587 29179 leader_election.cc:302] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:34.857906 31244 raft_consensus.cc:2781] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.858007 31244 raft_consensus.cc:482] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.858107 31244 raft_consensus.cc:3037] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.860088 31244 raft_consensus.cc:504] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.860646 31244 leader_election.cc:288] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:34.861061 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60527f81f34b45c199bf52bf0df53ee2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:34.861135 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60527f81f34b45c199bf52bf0df53ee2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:34.861256 29636 raft_consensus.cc:3037] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.861366 29504 raft_consensus.cc:3037] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.863237 29636 raft_consensus.cc:2454] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:34.863355 29504 raft_consensus.cc:2454] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:34.863863 29180 leader_election.cc:302] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:34.864106 31244 raft_consensus.cc:2781] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.864286 31244 raft_consensus.cc:686] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:34.864606 31244 consensus_queue.cc:237] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.867952 29096 catalog_manager.cc:5484] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:34.980645 31572 raft_consensus.cc:482] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.980825 31572 raft_consensus.cc:504] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.981681 31572 leader_election.cc:288] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:34.982089 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:34.982184 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:34.982378 29246 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:34.982405 29375 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:34.982810 29440 leader_election.cc:302] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:34.983093 31572 raft_consensus.cc:2781] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.983201 31572 raft_consensus.cc:482] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.983291 31572 raft_consensus.cc:3037] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.985428 31572 raft_consensus.cc:504] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.986001 31572 leader_election.cc:288] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:34.986721 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:34.986747 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:34.986912 29375 raft_consensus.cc:3037] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.986999 29246 raft_consensus.cc:3037] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.989109 29246 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:34.989120 29375 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:34.989528 29440 leader_election.cc:302] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:34.989905 31572 raft_consensus.cc:2781] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.990048 31572 raft_consensus.cc:686] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:34.990334 31572 consensus_queue.cc:237] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:34.994048 29096 catalog_manager.cc:5484] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:35.193599 31244 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.213678 31654 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.386809 31654 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.393257 31655 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:35.404054 31650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:35.404418 31650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:00:35.412403 31597 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:35.413586 31650 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:35.418282 31572 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
INSERT report
rows total: 2000
time total: 244.191 ms
time per row: 0.122096 ms
I20240501 14:00:35.856704 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:35.865895 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 37988c87ab494486be42cc398f307830 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.866400 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37988c87ab494486be42cc398f307830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.866631 29356 tablet_service.cc:1467] Processing CreateTablet for tablet bb1805e53a764f8db7d88a19b585bccf (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.866833 29485 tablet_service.cc:1467] Processing CreateTablet for tablet bb1805e53a764f8db7d88a19b585bccf (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.866956 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1805e53a764f8db7d88a19b585bccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.867221 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1805e53a764f8db7d88a19b585bccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.872004 31711 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:35.873769 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 37988c87ab494486be42cc398f307830 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.874258 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37988c87ab494486be42cc398f307830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.874922 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.875272 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623fd5c168d34a93a3dd77249aa9a391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.875285 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.875849 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623fd5c168d34a93a3dd77249aa9a391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.877431 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 37988c87ab494486be42cc398f307830 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.877667 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.877828 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37988c87ab494486be42cc398f307830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.878012 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623fd5c168d34a93a3dd77249aa9a391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.874167 29617 tablet_service.cc:1467] Processing CreateTablet for tablet bb1805e53a764f8db7d88a19b585bccf (DEFAULT_TABLE table=rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.881644 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1805e53a764f8db7d88a19b585bccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.885581 31713 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:35.888180 31713 tablet_bootstrap.cc:654] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.890985 31713 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:35.891198 31713 ts_tablet_manager.cc:1367] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 14:00:35.892472 31713 raft_consensus.cc:348] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.892742 31713 raft_consensus.cc:374] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.892854 31713 raft_consensus.cc:729] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.893177 31713 consensus_queue.cc:260] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.893971 31713 ts_tablet_manager.cc:1398] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:35.898108 31714 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:35.899209 31713 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:35.899718 31714 tablet_bootstrap.cc:654] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.900794 31711 tablet_bootstrap.cc:654] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.903012 31713 tablet_bootstrap.cc:654] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.903517 31714 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:35.903651 31714 ts_tablet_manager.cc:1367] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:00:35.904433 31714 raft_consensus.cc:348] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.904611 31714 raft_consensus.cc:374] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.904948 31713 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:35.905189 31713 ts_tablet_manager.cc:1367] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:00:35.905709 31714 raft_consensus.cc:729] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.905977 31713 raft_consensus.cc:348] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.906216 31713 raft_consensus.cc:374] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.906324 31713 raft_consensus.cc:729] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.906288 31714 consensus_queue.cc:260] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.906603 31713 consensus_queue.cc:260] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.907130 31713 ts_tablet_manager.cc:1398] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:35.907246 31714 ts_tablet_manager.cc:1398] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:35.907925 31714 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:35.908224 31711 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:35.908385 31711 ts_tablet_manager.cc:1367] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240501 14:00:35.909221 31711 raft_consensus.cc:348] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.909480 31711 raft_consensus.cc:374] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.909595 31711 raft_consensus.cc:729] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.909816 31714 tablet_bootstrap.cc:654] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.910048 31711 consensus_queue.cc:260] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.910677 31711 ts_tablet_manager.cc:1398] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:35.911036 31711 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:35.912575 31711 tablet_bootstrap.cc:654] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.913754 31712 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:35.915639 31712 tablet_bootstrap.cc:654] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.916045 31714 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:35.916407 31714 ts_tablet_manager.cc:1367] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:00:35.917281 31714 raft_consensus.cc:348] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.917433 31711 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:35.917589 31714 raft_consensus.cc:374] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.917640 31711 ts_tablet_manager.cc:1367] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:35.917757 31714 raft_consensus.cc:729] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.918004 31714 consensus_queue.cc:260] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.918416 31711 raft_consensus.cc:348] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.918603 31714 ts_tablet_manager.cc:1398] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:35.918706 31711 raft_consensus.cc:374] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.918785 31711 raft_consensus.cc:729] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.919070 31711 consensus_queue.cc:260] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.919713 31711 ts_tablet_manager.cc:1398] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:35.919744 31714 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:35.920763 31711 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:35.921447 31714 tablet_bootstrap.cc:654] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.923758 31711 tablet_bootstrap.cc:654] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.924623 31714 tablet_bootstrap.cc:492] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:35.924784 31714 ts_tablet_manager.cc:1367] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:35.926034 31714 raft_consensus.cc:348] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.926245 31712 tablet_bootstrap.cc:492] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:35.926263 31714 raft_consensus.cc:374] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.926457 31714 raft_consensus.cc:729] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.926527 31712 ts_tablet_manager.cc:1367] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:00:35.926769 31714 consensus_queue.cc:260] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:35.927642 31714 ts_tablet_manager.cc:1398] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:35.927896 31712 raft_consensus.cc:348] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.928254 31712 raft_consensus.cc:374] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.928407 31712 raft_consensus.cc:729] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.928817 31712 consensus_queue.cc:260] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.932379 31712 ts_tablet_manager.cc:1398] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:35.932636 31711 tablet_bootstrap.cc:492] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:35.932796 31711 ts_tablet_manager.cc:1367] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:00:35.933998 31711 raft_consensus.cc:348] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.934201 31711 raft_consensus.cc:374] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.934285 31711 raft_consensus.cc:729] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.934547 31711 consensus_queue.cc:260] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:35.936447 31711 ts_tablet_manager.cc:1398] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:35.996604 31597 raft_consensus.cc:482] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.996815 31597 raft_consensus.cc:504] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:35.997561 31597 leader_election.cc:288] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:35.998099 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1805e53a764f8db7d88a19b585bccf" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:35.998071 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1805e53a764f8db7d88a19b585bccf" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:35.998310 29374 raft_consensus.cc:2454] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:35.998334 29637 raft_consensus.cc:2454] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:35.998756 29440 leader_election.cc:302] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:35.999042 31597 raft_consensus.cc:2781] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:35.999132 31597 raft_consensus.cc:482] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:35.999256 31597 raft_consensus.cc:3037] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.001513 31597 raft_consensus.cc:504] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:36.002509 31597 leader_election.cc:288] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:36.002602 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1805e53a764f8db7d88a19b585bccf" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:36.002827 29374 raft_consensus.cc:3037] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.002795 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1805e53a764f8db7d88a19b585bccf" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:36.003016 29637 raft_consensus.cc:3037] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.004719 29374 raft_consensus.cc:2454] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:36.005107 29442 leader_election.cc:302] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:36.005149 29637 raft_consensus.cc:2454] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:36.005410 31572 raft_consensus.cc:2781] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.005578 31572 raft_consensus.cc:686] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:36.005841 31572 consensus_queue.cc:237] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:36.009534 29103 catalog_manager.cc:5484] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:36.021844 31655 raft_consensus.cc:482] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.022044 31655 raft_consensus.cc:504] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:36.022477 31618 raft_consensus.cc:482] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.022620 31618 raft_consensus.cc:504] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:36.022789 31655 leader_election.cc:288] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:36.023206 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623fd5c168d34a93a3dd77249aa9a391" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:36.023358 31618 leader_election.cc:288] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:36.023461 29637 raft_consensus.cc:2454] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:36.023350 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623fd5c168d34a93a3dd77249aa9a391" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:36.023682 29374 raft_consensus.cc:2454] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:36.024050 29179 leader_election.cc:302] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:36.024139 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37988c87ab494486be42cc398f307830" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:36.024201 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37988c87ab494486be42cc398f307830" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:36.024359 29245 raft_consensus.cc:2454] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:36.024402 29374 raft_consensus.cc:2454] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:36.024525 31655 raft_consensus.cc:2781] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.024618 31655 raft_consensus.cc:482] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.024710 31655 raft_consensus.cc:3037] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.024879 29572 leader_election.cc:302] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:36.025336 31618 raft_consensus.cc:2781] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.025439 31618 raft_consensus.cc:482] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.025528 31618 raft_consensus.cc:3037] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.027334 31618 raft_consensus.cc:504] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:36.027565 31655 raft_consensus.cc:504] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:36.028213 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37988c87ab494486be42cc398f307830" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:36.028416 31618 leader_election.cc:288] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:36.028564 31655 leader_election.cc:288] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:36.028493 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623fd5c168d34a93a3dd77249aa9a391" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:36.029064 29375 raft_consensus.cc:3037] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.029086 29374 raft_consensus.cc:3037] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.029067 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623fd5c168d34a93a3dd77249aa9a391" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:36.029256 29637 raft_consensus.cc:3037] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.031039 29637 raft_consensus.cc:2454] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:36.031348 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37988c87ab494486be42cc398f307830" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:36.031421 29179 leader_election.cc:302] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:36.031512 29245 raft_consensus.cc:3037] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.031697 31655 raft_consensus.cc:2781] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.031847 31655 raft_consensus.cc:686] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:36.031898 29375 raft_consensus.cc:2454] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:36.032265 29374 raft_consensus.cc:2454] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:36.032234 31655 consensus_queue.cc:237] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:36.033201 29572 leader_election.cc:302] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:36.033468 31618 raft_consensus.cc:2781] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.034757 31618 raft_consensus.cc:686] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:36.034765 29245 raft_consensus.cc:2454] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:36.035149 31618 consensus_queue.cc:237] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:36.038895 29103 catalog_manager.cc:5484] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:36.051697 29103 catalog_manager.cc:5484] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:36.432209 31244 consensus_queue.cc:1035] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.521682 31655 consensus_queue.cc:1035] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.545264 31618 consensus_queue.cc:1035] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:36.551734 31728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:36.552172 31728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:00:36.557638 31572 consensus_queue.cc:1035] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:36.565274 31728 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:36.605355 31743 consensus_queue.cc:1035] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.612792 31597 consensus_queue.cc:1035] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 270.054 ms
time per row: 0.135027 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:38.072538 29104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:38.081931 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 500b9c9bdfa649bd883d6ef2d6e541fc (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.082424 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500b9c9bdfa649bd883d6ef2d6e541fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.082880 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 0e33e4f9b1204797aeff0bca36c93911 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.083242 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e33e4f9b1204797aeff0bca36c93911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.085867 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 500b9c9bdfa649bd883d6ef2d6e541fc (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.085966 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 0e33e4f9b1204797aeff0bca36c93911 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.086301 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500b9c9bdfa649bd883d6ef2d6e541fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.087388 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 500b9c9bdfa649bd883d6ef2d6e541fc (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.087832 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500b9c9bdfa649bd883d6ef2d6e541fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.088315 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e33e4f9b1204797aeff0bca36c93911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.088501 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 5893a54b753f4d52a900d34140f6363d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.088874 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5893a54b753f4d52a900d34140f6363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.091348 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 0e33e4f9b1204797aeff0bca36c93911 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.091763 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e33e4f9b1204797aeff0bca36c93911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.092481 29486 tablet_service.cc:1467] Processing CreateTablet for tablet 5893a54b753f4d52a900d34140f6363d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.092857 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5893a54b753f4d52a900d34140f6363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.097112 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 5893a54b753f4d52a900d34140f6363d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.097683 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5893a54b753f4d52a900d34140f6363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.107673 31800 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:38.108824 31798 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:38.109733 31800 tablet_bootstrap.cc:654] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.110785 31798 tablet_bootstrap.cc:654] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.110909 31799 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:38.112738 31801 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:38.113268 31799 tablet_bootstrap.cc:654] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.116636 31801 tablet_bootstrap.cc:654] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.121249 31798 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:38.121592 31798 ts_tablet_manager.cc:1367] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240501 14:00:38.121845 31800 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:38.122116 31800 ts_tablet_manager.cc:1367] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240501 14:00:38.122148 31799 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:38.122283 31799 ts_tablet_manager.cc:1367] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 14:00:38.122629 31798 raft_consensus.cc:348] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.122980 31798 raft_consensus.cc:374] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.123143 31801 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:38.123152 31798 raft_consensus.cc:729] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.123087 31799 raft_consensus.cc:348] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.123294 31799 raft_consensus.cc:374] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.123353 31801 ts_tablet_manager.cc:1367] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240501 14:00:38.123461 31799 raft_consensus.cc:729] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.123719 31799 consensus_queue.cc:260] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.123535 31798 consensus_queue.cc:260] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.124146 31800 raft_consensus.cc:348] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.124410 31799 ts_tablet_manager.cc:1398] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:38.124397 31800 raft_consensus.cc:374] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.124500 31800 raft_consensus.cc:729] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.124595 31801 raft_consensus.cc:348] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.124758 31799 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:38.124758 31800 consensus_queue.cc:260] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.124891 31801 raft_consensus.cc:374] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.125294 31801 raft_consensus.cc:729] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.125329 31800 ts_tablet_manager.cc:1398] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:38.125614 31801 consensus_queue.cc:260] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.126479 31800 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:38.126524 31801 ts_tablet_manager.cc:1398] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:38.126940 31801 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:38.127115 31799 tablet_bootstrap.cc:654] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.128233 31800 tablet_bootstrap.cc:654] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.128949 31801 tablet_bootstrap.cc:654] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.137305 31800 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:38.137609 31800 ts_tablet_manager.cc:1367] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 14:00:38.138793 31800 raft_consensus.cc:348] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.139791 31800 raft_consensus.cc:374] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.139967 31800 raft_consensus.cc:729] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.140257 31800 consensus_queue.cc:260] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.143711 31800 ts_tablet_manager.cc:1398] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:38.144184 31800 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:38.147233 31798 ts_tablet_manager.cc:1398] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:00:38.148918 31800 tablet_bootstrap.cc:654] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.152139 31798 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:38.153653 31801 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:38.154419 31801 ts_tablet_manager.cc:1367] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.005s
I20240501 14:00:38.154608 31798 tablet_bootstrap.cc:654] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.155678 31801 raft_consensus.cc:348] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.158905 31800 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:38.159196 31800 ts_tablet_manager.cc:1367] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240501 14:00:38.160209 31800 raft_consensus.cc:348] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.160611 31800 raft_consensus.cc:374] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.160799 31800 raft_consensus.cc:729] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.161164 31800 consensus_queue.cc:260] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.163017 31798 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:38.163734 31800 ts_tablet_manager.cc:1398] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:38.163784 31798 ts_tablet_manager.cc:1367] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240501 14:00:38.167599 31801 raft_consensus.cc:374] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.167768 31801 raft_consensus.cc:729] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.168089 31801 consensus_queue.cc:260] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.168752 31801 ts_tablet_manager.cc:1398] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:00:38.164757 31798 raft_consensus.cc:348] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.170579 31798 raft_consensus.cc:374] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.170686 31798 raft_consensus.cc:729] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.170948 31798 consensus_queue.cc:260] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.173621 31798 ts_tablet_manager.cc:1398] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:00:38.175328 31799 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:38.175535 31799 ts_tablet_manager.cc:1367] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.018s
I20240501 14:00:38.176416 31799 raft_consensus.cc:348] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.176726 31799 raft_consensus.cc:374] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.176846 31799 raft_consensus.cc:729] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.177162 31799 consensus_queue.cc:260] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.177771 31799 ts_tablet_manager.cc:1398] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:38.319612 31792 raft_consensus.cc:482] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.319828 31792 raft_consensus.cc:504] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.320637 31792 leader_election.cc:288] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:38.321055 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e33e4f9b1204797aeff0bca36c93911" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:38.321120 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e33e4f9b1204797aeff0bca36c93911" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:38.321235 29505 raft_consensus.cc:2454] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:38.321400 29637 raft_consensus.cc:2454] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:38.321679 29311 leader_election.cc:302] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:38.321714 31734 raft_consensus.cc:482] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.321928 31792 raft_consensus.cc:2781] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.321869 31734 raft_consensus.cc:504] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.322042 31792 raft_consensus.cc:482] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.322156 31792 raft_consensus.cc:3037] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.322659 31734 leader_election.cc:288] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:38.323112 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:38.323311 29505 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:38.323323 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:38.323549 29375 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:38.323886 29180 leader_election.cc:302] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:38.324471 31734 raft_consensus.cc:2781] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.324626 31734 raft_consensus.cc:482] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.324759 31734 raft_consensus.cc:3037] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.325462 31792 raft_consensus.cc:504] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.326013 31792 leader_election.cc:288] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:38.326463 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e33e4f9b1204797aeff0bca36c93911" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:38.326593 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e33e4f9b1204797aeff0bca36c93911" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:38.326720 29637 raft_consensus.cc:3037] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.326784 29505 raft_consensus.cc:3037] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.327136 31734 raft_consensus.cc:504] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.328295 31734 leader_election.cc:288] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:38.328823 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:38.329041 29375 raft_consensus.cc:3037] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.329288 29637 raft_consensus.cc:2454] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:38.329308 29505 raft_consensus.cc:2454] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:38.329610 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:38.329681 29310 leader_election.cc:302] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:38.329782 29506 raft_consensus.cc:3037] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.329977 31792 raft_consensus.cc:2781] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.330130 31792 raft_consensus.cc:686] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:38.330538 31792 consensus_queue.cc:237] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.331624 29375 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:38.332031 29181 leader_election.cc:302] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:38.332281 31244 raft_consensus.cc:2781] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.332341 29506 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:38.332449 31244 raft_consensus.cc:686] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:38.332697 31244 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:38.335892 29103 catalog_manager.cc:5484] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:38.340799 29103 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:38.545132 31790 raft_consensus.cc:482] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.545331 31790 raft_consensus.cc:504] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.545936 31790 leader_election.cc:288] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:38.546456 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:38.546484 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:38.546676 29246 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:38.546689 29637 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:00:38.547103 29440 leader_election.cc:302] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:38.547365 31790 raft_consensus.cc:2781] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.547464 31790 raft_consensus.cc:482] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.547552 31790 raft_consensus.cc:3037] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.549463 31790 raft_consensus.cc:504] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.550076 31790 leader_election.cc:288] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:38.550557 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:38.550618 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:38.550791 29246 raft_consensus.cc:3037] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.550805 29637 raft_consensus.cc:3037] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.553682 29637 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:38.553682 29246 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:00:38.554165 29440 leader_election.cc:302] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:38.554482 31790 raft_consensus.cc:2781] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.554632 31790 raft_consensus.cc:686] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:00:38.554911 31790 consensus_queue.cc:237] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:38.558845 29103 catalog_manager.cc:5484] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:38.877779 31792 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.883033 31792 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.923336 31734 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.934931 31734 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.996603 31812 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.001397 31812 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:39.070833 31817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:39.071166 31817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:39.079422 31817 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
INSERT report
rows total: 2000
time total: 306.484 ms
time per row: 0.153242 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:40.563848 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:32783
--local_ip_for_outbound_sockets=127.22.33.69
--tserver_master_addrs=127.22.33.126:35223
--webserver_port=34427
--webserver_interface=127.22.33.69
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:00:40.935883 31881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:40.936201 31881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:40.936331 31881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:40.946117 31881 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:40.946265 31881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:40.946450 31881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 14:00:40.956789 31881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:32783
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=34427
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:40.958204 31881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:40.960186 31881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:40.973898 31889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:40.974601 31888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:40.975327 31891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:40.976212 31881 server_base.cc:1034] running on GCE node
I20240501 14:00:40.977025 31881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:40.978277 31881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:40.979503 31881 hybrid_clock.cc:648] HybridClock initialized: now 1714572040979451 us; error 46 us; skew 500 ppm
I20240501 14:00:40.979835 31881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:40.982335 31881 webserver.cc:460] Webserver started at http://127.22.33.69:34427/ using document root <none> and password file <none>
I20240501 14:00:40.983184 31881 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:40.983362 31881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:40.989897 31881 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:40.993127 31897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:40.994784 31881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:40.994987 31881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9903ed35bca144bcb74e9bf5102117e7"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-3610"
I20240501 14:00:40.995944 31881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:41.028177 31881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:41.029099 31881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:41.029397 31881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:41.030469 31881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:41.032058 31881 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:41.032140 31881 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:41.032274 31881 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:41.032332 31881 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:41.075296 31881 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:32783
I20240501 14:00:41.075335 32009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:32783 every 8 connection(s)
I20240501 14:00:41.076905 31881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:41.077737 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 31881
I20240501 14:00:41.078511 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:37745
--local_ip_for_outbound_sockets=127.22.33.70
--tserver_master_addrs=127.22.33.126:35223
--webserver_port=43603
--webserver_interface=127.22.33.70
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:41.091912 32010 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:41.092319 32010 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:41.093654 32010 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:41.095395 29103 ts_manager.cc:194] Re-registered known tserver with Master: 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:41.096357 29103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:51707
W20240501 14:00:41.460594 32014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:41.460902 32014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:41.461095 32014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:41.469264 32014 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:41.469408 32014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:41.469570 32014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 14:00:41.483901 32014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:37745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=43603
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:41.485633 32014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:41.487900 32014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:41.503190 32020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:41.503706 32021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:41.504566 32023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:41.505385 32014 server_base.cc:1034] running on GCE node
I20240501 14:00:41.506050 32014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:41.507083 32014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:41.508334 32014 hybrid_clock.cc:648] HybridClock initialized: now 1714572041508278 us; error 38 us; skew 500 ppm
I20240501 14:00:41.508690 32014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:41.511814 32014 webserver.cc:460] Webserver started at http://127.22.33.70:43603/ using document root <none> and password file <none>
I20240501 14:00:41.512820 32014 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:41.512986 32014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:41.518644 32014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 14:00:41.521579 32029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:41.522965 32014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:41.523123 32014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0a697f3a1ee544daa7073814ab845d58"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:41.523947 32014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:41.560325 32014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:41.561272 32014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:41.561556 32014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:41.562541 32014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:41.564131 32014 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:41.564213 32014 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:41.564347 32014 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:41.564406 32014 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:41.605141 32014 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:37745
I20240501 14:00:41.605306 32141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:37745 every 8 connection(s)
I20240501 14:00:41.606864 32014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:00:41.607405 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 32014
I20240501 14:00:41.608224 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:38743
--local_ip_for_outbound_sockets=127.22.33.71
--tserver_master_addrs=127.22.33.126:35223
--webserver_port=34479
--webserver_interface=127.22.33.71
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:41.623957 32142 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:41.624302 32142 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:41.625145 32142 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:41.626955 29103 ts_manager.cc:194] Re-registered known tserver with Master: 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:41.627971 29103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:56127
W20240501 14:00:41.987004 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:41.987426 32146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:41.987581 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:41.995808 32146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:41.995949 32146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:41.996057 32146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 14:00:42.006520 32146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:38743
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=34479
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:42.007881 32146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:42.009907 32146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:42.026648 32152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:42.027740 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:42.028544 32154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:42.029165 32146 server_base.cc:1034] running on GCE node
I20240501 14:00:42.030115 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:42.031241 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:42.032461 32146 hybrid_clock.cc:648] HybridClock initialized: now 1714572042032395 us; error 55 us; skew 500 ppm
I20240501 14:00:42.032820 32146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:42.035415 32146 webserver.cc:460] Webserver started at http://127.22.33.71:34479/ using document root <none> and password file <none>
I20240501 14:00:42.036258 32146 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:42.036402 32146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:42.042172 32146 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 14:00:42.045205 32160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.046623 32146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:42.046772 32146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6a446ac802a7457d9f27076781f21d79"
format_stamp: "Formatted at 2024-05-01 14:00:07 on dist-test-slave-3610"
I20240501 14:00:42.047706 32146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:42.063699 32146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:42.064690 32146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:42.064960 32146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:42.065924 32146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:42.067397 32146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:42.067468 32146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.067593 32146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:42.067643 32146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.099370 32010 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
I20240501 14:00:42.114411 32146 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:38743
I20240501 14:00:42.114423 32273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:38743 every 8 connection(s)
I20240501 14:00:42.116097 32146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:00:42.125324 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 32146
I20240501 14:00:42.126078 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.72:37615
--local_ip_for_outbound_sockets=127.22.33.72
--tserver_master_addrs=127.22.33.126:35223
--webserver_port=44653
--webserver_interface=127.22.33.72
--builtin_ntp_servers=127.22.33.84:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:42.130631 32274 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:42.130998 32274 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:42.131935 32274 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:42.133440 29103 ts_manager.cc:194] Re-registered known tserver with Master: 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:42.134217 29103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:38019
W20240501 14:00:42.520651 32277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:42.520992 32277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:42.521116 32277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:42.529548 32277 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:42.529685 32277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:42.529839 32277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.72
I20240501 14:00:42.539902 32277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.72:37615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.33.72
--webserver_port=44653
--tserver_master_addrs=127.22.33.126:35223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.72
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:42.541518 32277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:42.543309 32277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:42.556994 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:42.557655 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:42.558504 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:42.559362 32277 server_base.cc:1034] running on GCE node
I20240501 14:00:42.559973 32277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:42.560995 32277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:42.562247 32277 hybrid_clock.cc:648] HybridClock initialized: now 1714572042562177 us; error 57 us; skew 500 ppm
I20240501 14:00:42.562618 32277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:42.565318 32277 webserver.cc:460] Webserver started at http://127.22.33.72:44653/ using document root <none> and password file <none>
I20240501 14:00:42.566076 32277 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:42.566206 32277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:42.571902 32277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:00:42.574702 32293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.576045 32277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:42.576191 32277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6edd6a3430d44c38a19c2804e555059f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-3610"
I20240501 14:00:42.577025 32277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:42.598758 32277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:42.599606 32277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:42.599918 32277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:42.601022 32277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:42.602950 32277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:42.603032 32277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.603195 32277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:42.603281 32277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:42.631402 32142 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
I20240501 14:00:42.657584 32277 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.72:37615
I20240501 14:00:42.657613 32407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.72:37615 every 8 connection(s)
I20240501 14:00:42.659577 32277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:00:42.666247 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 32277
I20240501 14:00:42.674398 32408 heartbeater.cc:347] Connected to a master server at 127.22.33.126:35223
I20240501 14:00:42.674793 32408 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:42.675791 32408 heartbeater.cc:511] Master 127.22.33.126:35223 requested a full tablet report, sending...
I20240501 14:00:42.677723 29098 ts_manager.cc:194] Re-registered known tserver with Master: 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:42.677726 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:42.678681 29098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.72:44561
I20240501 14:00:42.722828 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b002ebf0448f9d45a3f07209059b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]), partition=
I20240501 14:00:42.723938 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b002ebf0448f9d45a3f07209059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.738009 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b002ebf0448f9d45a3f07209059b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]), partition=
I20240501 14:00:42.741405 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b002ebf0448f9d45a3f07209059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.751013 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b002ebf0448f9d45a3f07209059b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]), partition=
I20240501 14:00:42.752326 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b002ebf0448f9d45a3f07209059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.753149 32426 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:42.758282 32426 tablet_bootstrap.cc:654] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.760033 32426 log.cc:824] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:42.766265 32427 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:42.772646 32427 tablet_bootstrap.cc:654] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.775544 32429 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:42.775607 32426 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:42.775918 32426 ts_tablet_manager.cc:1367] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.004s
I20240501 14:00:42.777267 32427 log.cc:824] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:42.780517 32429 tablet_bootstrap.cc:654] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.782138 32429 log.cc:824] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:42.788506 32427 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:42.788403 32426 raft_consensus.cc:348] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.788929 32427 ts_tablet_manager.cc:1367] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.009s
I20240501 14:00:42.788987 32426 raft_consensus.cc:374] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.789144 32426 raft_consensus.cc:729] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.790043 32426 consensus_queue.cc:260] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.791239 32429 tablet_bootstrap.cc:492] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:42.791603 32429 ts_tablet_manager.cc:1367] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240501 14:00:42.800361 32429 raft_consensus.cc:348] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.800083 32427 raft_consensus.cc:348] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.800917 32429 raft_consensus.cc:374] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.801038 32429 raft_consensus.cc:729] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.801146 32427 raft_consensus.cc:374] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.801319 32427 raft_consensus.cc:729] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.801928 32429 consensus_queue.cc:260] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.802135 32427 consensus_queue.cc:260] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.804304 32426 ts_tablet_manager.cc:1398] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.028s user 0.013s sys 0.007s
I20240501 14:00:42.810235 32429 ts_tablet_manager.cc:1398] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20240501 14:00:42.810667 32274 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
I20240501 14:00:42.811703 32408 heartbeater.cc:503] Master 127.22.33.126:35223 was elected leader, sending a full tablet report...
I20240501 14:00:42.812489 32427 ts_tablet_manager.cc:1398] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
W20240501 14:00:42.829799 32011 tablet.cc:2369] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:42.867477 32434 raft_consensus.cc:482] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:42.867767 32434 raft_consensus.cc:504] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
W20240501 14:00:42.868129 32275 tablet.cc:2369] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:42.869613 32434 leader_election.cc:288] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:42.878919 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b002ebf0448f9d45a3f07209059b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:42.879487 31965 raft_consensus.cc:2454] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:42.879793 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b002ebf0448f9d45a3f07209059b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:42.880211 32296 leader_election.cc:302] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:42.880308 32227 raft_consensus.cc:2454] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:42.880654 32434 raft_consensus.cc:2781] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:42.880833 32434 raft_consensus.cc:482] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:42.880951 32434 raft_consensus.cc:3037] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.883476 32434 raft_consensus.cc:504] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.884027 32434 leader_election.cc:288] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:42.884717 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b002ebf0448f9d45a3f07209059b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:42.884794 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b002ebf0448f9d45a3f07209059b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:42.884918 32227 raft_consensus.cc:3037] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.885040 31965 raft_consensus.cc:3037] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.887248 31965 raft_consensus.cc:2454] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:42.887853 32296 leader_election.cc:302] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:42.888644 32227 raft_consensus.cc:2454] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:42.889003 32434 raft_consensus.cc:2781] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:42.890031 32434 raft_consensus.cc:686] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:42.890529 32434 consensus_queue.cc:237] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:42.895577 29103 catalog_manager.cc:5484] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:00:42.911477 32409 tablet.cc:2369] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:43.043812 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:43.054172 31965 raft_consensus.cc:1261] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:43.055684 32227 raft_consensus.cc:1261] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:43.056905 32434 consensus_queue.cc:1035] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.057288 32437 consensus_queue.cc:1035] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.070626 29098 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]). Expected version 1 got 0
I20240501 14:00:43.073205 32440 mvcc.cc:205] Tried to move back new op lower bound from 7022887088335486976 to 7022887087683567616. Current Snapshot: MvccSnapshot[applied={T|T < 7022887088335486976}]
I20240501 14:00:43.075363 32445 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:43.079989 32445 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:43.089740 32446 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:43.091584 32446 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:43.092460 29098 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]). Expected version 1 got 0
I20240501 14:00:43.093478 29103 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]). Expected version 1 got 0
I20240501 14:00:43.100230 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5] alter complete (version 1)
I20240501 14:00:43.103756 32447 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:43.105397 32447 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:43.105476 29098 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5]). Expected version 1 got 0
I20240501 14:00:43.168618 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:00:43.168718 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:43.168916 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:00:43.173986 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 1c5ce76b9ac34f69bb79049abecfb674
I20240501 14:00:43.175931 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.175966 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:43.176256 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet d201cef3c1164bdeb443b3f75ffcb076
I20240501 14:00:43.176640 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ce76b9ac34f69bb79049abecfb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:43.177222 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.178928 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:43.183792 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet f8ff1d95887a4e2c9b1b2ff84fb45089
I20240501 14:00:43.183863 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet d201cef3c1164bdeb443b3f75ffcb076 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:43.184659 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:43.186960 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:43.187767 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ff1d95887a4e2c9b1b2ff84fb45089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.188596 32462 tablet_replica.cc:330] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:43.189170 32462 raft_consensus.cc:2229] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.189589 32462 raft_consensus.cc:2258] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.191955 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 6c86d880c077411a99e6b9597adeeb0d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.192690 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c86d880c077411a99e6b9597adeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.192935 32461 tablet_replica.cc:330] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:43.193416 32461 raft_consensus.cc:2229] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.193750 32461 raft_consensus.cc:2258] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.195358 32461 ts_tablet_manager.cc:1875] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.197466 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 414c148db46543b786bfd8b0faecfe96 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.197764 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 6c86d880c077411a99e6b9597adeeb0d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.198041 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414c148db46543b786bfd8b0faecfe96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.198194 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c86d880c077411a99e6b9597adeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.201108 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.201608 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.201462 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.201915 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.204800 32460 tablet_replica.cc:330] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:43.205317 32460 raft_consensus.cc:2229] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.205466 32429 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:43.205710 32460 raft_consensus.cc:2258] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.206277 32459 tablet_replica.cc:330] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.206496 32464 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:43.206872 32459 raft_consensus.cc:2229] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.206910 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:43.207360 32429 tablet_bootstrap.cc:654] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.207542 32459 raft_consensus.cc:2258] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.208551 32464 tablet_bootstrap.cc:654] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.210876 32460 ts_tablet_manager.cc:1875] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.213485 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 414c148db46543b786bfd8b0faecfe96 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.214183 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414c148db46543b786bfd8b0faecfe96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.219343 32464 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:43.219692 32464 ts_tablet_manager.cc:1367] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 14:00:43.222437 32461 ts_tablet_manager.cc:1888] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.222615 32461 log.cc:1196] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c5ce76b9ac34f69bb79049abecfb674
I20240501 14:00:43.223187 32461 ts_tablet_manager.cc:1909] T 1c5ce76b9ac34f69bb79049abecfb674 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:43.220932 32464 raft_consensus.cc:348] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.224825 32461 tablet_replica.cc:330] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:43.224936 32464 raft_consensus.cc:374] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.225000 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 1c5ce76b9ac34f69bb79049abecfb674 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.225140 32464 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.225481 32459 ts_tablet_manager.cc:1875] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.225445 32464 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.226006 32464 ts_tablet_manager.cc:1398] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:43.227510 32468 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:43.231588 32468 tablet_bootstrap.cc:654] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.231653 32461 raft_consensus.cc:2229] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.232204 32461 raft_consensus.cc:2258] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.232923 32461 ts_tablet_manager.cc:1875] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.234782 32460 ts_tablet_manager.cc:1888] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.234957 32460 log.cc:1196] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c5ce76b9ac34f69bb79049abecfb674
I20240501 14:00:43.236835 32445 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.237344 32445 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:43.237684 32460 ts_tablet_manager.cc:1909] T 1c5ce76b9ac34f69bb79049abecfb674 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:43.239485 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 1c5ce76b9ac34f69bb79049abecfb674 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.240351 32446 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.240875 32446 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:43.242391 32447 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.242846 32447 tablet.cc:1620] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:43.243068 32460 tablet_replica.cc:330] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:43.243295 32460 raft_consensus.cc:2229] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.243531 32460 raft_consensus.cc:2258] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.244241 32460 ts_tablet_manager.cc:1875] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.252434 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=8b528b4859b54372a6290061b12808c5] alter complete (version 2)
I20240501 14:00:43.258683 32461 ts_tablet_manager.cc:1888] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.258708 32459 ts_tablet_manager.cc:1888] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.258857 32461 log.cc:1196] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8ff1d95887a4e2c9b1b2ff84fb45089
I20240501 14:00:43.258857 32459 log.cc:1196] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c5ce76b9ac34f69bb79049abecfb674
I20240501 14:00:43.259311 32461 ts_tablet_manager.cc:1909] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:43.259444 32459 ts_tablet_manager.cc:1909] T 1c5ce76b9ac34f69bb79049abecfb674 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.261198 32459 tablet_replica.cc:330] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.261487 32459 raft_consensus.cc:2229] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.261881 32459 raft_consensus.cc:2258] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.262925 32459 ts_tablet_manager.cc:1875] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.271751 32459 ts_tablet_manager.cc:1888] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.272001 32459 log.cc:1196] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/d201cef3c1164bdeb443b3f75ffcb076
I20240501 14:00:43.272480 32459 ts_tablet_manager.cc:1909] T d201cef3c1164bdeb443b3f75ffcb076 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.273592 32459 tablet_replica.cc:330] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.273861 32459 raft_consensus.cc:2229] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.274209 32459 raft_consensus.cc:2258] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.274420 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.275158 32459 ts_tablet_manager.cc:1875] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.276064 32460 ts_tablet_manager.cc:1888] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.277876 32460 log.cc:1196] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/d201cef3c1164bdeb443b3f75ffcb076
I20240501 14:00:43.278594 32460 ts_tablet_manager.cc:1909] T d201cef3c1164bdeb443b3f75ffcb076 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:43.283449 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet d201cef3c1164bdeb443b3f75ffcb076 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.284539 32426 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:43.285894 32459 ts_tablet_manager.cc:1888] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.286124 32459 log.cc:1196] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8ff1d95887a4e2c9b1b2ff84fb45089
I20240501 14:00:43.286352 32426 tablet_bootstrap.cc:654] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.286708 32459 ts_tablet_manager.cc:1909] T f8ff1d95887a4e2c9b1b2ff84fb45089 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.295233 32462 ts_tablet_manager.cc:1875] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:00:43.298753 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 88636.6us
I20240501 14:00:43.299360 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 1c5ce76b9ac34f69bb79049abecfb674 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.299674 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet d201cef3c1164bdeb443b3f75ffcb076 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.299870 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.302699 32429 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:43.302884 32429 ts_tablet_manager.cc:1367] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.098s user 0.018s sys 0.012s
I20240501 14:00:43.303956 32426 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:43.304131 32426 ts_tablet_manager.cc:1367] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20240501 14:00:43.304179 32429 raft_consensus.cc:348] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.304574 32429 raft_consensus.cc:374] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.304672 32429 raft_consensus.cc:729] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.304920 32429 consensus_queue.cc:260] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.305471 32429 ts_tablet_manager.cc:1398] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:43.306728 32426 raft_consensus.cc:348] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.306970 32426 raft_consensus.cc:374] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.307082 32426 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.307411 32426 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.307792 32462 ts_tablet_manager.cc:1888] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.307974 32462 log.cc:1196] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/d201cef3c1164bdeb443b3f75ffcb076
I20240501 14:00:43.308511 32462 ts_tablet_manager.cc:1909] T d201cef3c1164bdeb443b3f75ffcb076 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:43.310282 32462 tablet_replica.cc:330] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:43.310583 32462 raft_consensus.cc:2229] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.310850 32462 raft_consensus.cc:2258] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.310920 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet d201cef3c1164bdeb443b3f75ffcb076 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.311422 32432 raft_consensus.cc:482] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.311736 32462 ts_tablet_manager.cc:1875] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.311601 32432 raft_consensus.cc:504] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.313640 32432 leader_election.cc:288] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:43.319682 32426 ts_tablet_manager.cc:1398] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240501 14:00:43.320078 32426 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:43.323020 32468 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:43.323120 32426 tablet_bootstrap.cc:654] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.323199 32468 ts_tablet_manager.cc:1367] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.096s user 0.016s sys 0.025s
I20240501 14:00:43.324256 32468 raft_consensus.cc:348] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.324522 32468 raft_consensus.cc:374] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.324604 32468 raft_consensus.cc:729] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.324896 32468 consensus_queue.cc:260] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.325649 32468 ts_tablet_manager.cc:1398] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:43.327334 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 6c86d880c077411a99e6b9597adeeb0d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.328787 32462 ts_tablet_manager.cc:1888] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.328923 32462 log.cc:1196] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8ff1d95887a4e2c9b1b2ff84fb45089
I20240501 14:00:43.329286 32462 ts_tablet_manager.cc:1909] T f8ff1d95887a4e2c9b1b2ff84fb45089 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:43.330449 32075 tablet_service.cc:1467] Processing CreateTablet for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.333822 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet f8ff1d95887a4e2c9b1b2ff84fb45089 (table rebalance_test_table_0 [id=874aebd6ef644614be63d85b8082f56a]) successfully deleted
I20240501 14:00:43.334359 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 414c148db46543b786bfd8b0faecfe96 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.335445 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c86d880c077411a99e6b9597adeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.336602 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f354a7a3c4f4c61a62f2c3e0cbbf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.337460 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414c148db46543b786bfd8b0faecfe96. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:43.340530 31900 leader_election.cc:341] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Not found: Tablet not found: 6c86d880c077411a99e6b9597adeeb0d
I20240501 14:00:43.347688 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:43.347965 29245 raft_consensus.cc:2454] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:43.348639 31898 leader_election.cc:302] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:43.349730 32432 raft_consensus.cc:2781] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:43.349890 32432 raft_consensus.cc:482] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:43.350004 32432 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.353510 32432 raft_consensus.cc:504] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.355752 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:43.355976 29245 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:43.357259 31900 leader_election.cc:341] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Not found: Tablet not found: 6c86d880c077411a99e6b9597adeeb0d
I20240501 14:00:43.358816 32426 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:43.358995 32426 ts_tablet_manager.cc:1367] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.011s
I20240501 14:00:43.359905 29245 raft_consensus.cc:2454] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:43.359869 32426 raft_consensus.cc:348] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.360196 32426 raft_consensus.cc:374] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.360296 32426 raft_consensus.cc:729] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.360707 32426 consensus_queue.cc:260] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.360862 31898 leader_election.cc:302] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:43.361495 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:43.361706 32432 leader_election.cc:288] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:43.362260 32432 raft_consensus.cc:2781] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:43.362470 32432 raft_consensus.cc:686] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:43.363055 32432 consensus_queue.cc:237] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.368218 32426 ts_tablet_manager.cc:1398] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 14:00:43.368657 32426 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:43.371995 32426 tablet_bootstrap.cc:654] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.378767 29103 catalog_manager.cc:5484] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
W20240501 14:00:43.380946 29553 tablet.cc:2369] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:43.385771 32432 raft_consensus.cc:482] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.385993 32432 raft_consensus.cc:504] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.386710 32445 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.387732 32432 leader_election.cc:288] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:43.389485 32447 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.401719 32483 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
W20240501 14:00:43.402678 31900 leader_election.cc:341] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Not found: Tablet not found: 3f354a7a3c4f4c61a62f2c3e0cbbf812
I20240501 14:00:43.408318 32483 tablet_bootstrap.cc:654] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.411819 32426 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:43.412882 32426 ts_tablet_manager.cc:1367] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.018s
I20240501 14:00:43.414101 32426 raft_consensus.cc:348] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.414953 32483 log.cc:824] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:43.415758 32426 raft_consensus.cc:374] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.415997 32426 raft_consensus.cc:729] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.416416 32426 consensus_queue.cc:260] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.417310 32426 ts_tablet_manager.cc:1398] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:43.419044 32446 tablet.cc:1722] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.431360 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=8b528b4859b54372a6290061b12808c5] alter complete (version 3)
I20240501 14:00:43.434516 32279 raft_consensus.cc:482] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.434707 32279 raft_consensus.cc:504] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.435606 32279 leader_election.cc:288] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:43.453423 32483 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:43.453769 32483 ts_tablet_manager.cc:1367] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.012s
I20240501 14:00:43.455991 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:43.456260 29504 raft_consensus.cc:2454] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:43.471997 31899 leader_election.cc:302] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:43.472344 32432 raft_consensus.cc:2781] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:43.472453 32432 raft_consensus.cc:482] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:43.472558 32432 raft_consensus.cc:3037] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:43.474443 29292 tablet.cc:2369] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:43.475406 32432 raft_consensus.cc:504] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.476559 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
W20240501 14:00:43.477586 31900 leader_election.cc:341] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:43.477947 32432 leader_election.cc:288] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:43.479190 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:43.479414 29504 raft_consensus.cc:3037] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.480515 32483 raft_consensus.cc:348] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.480950 32483 raft_consensus.cc:374] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.481083 32483 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.481962 32483 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.483325 29504 raft_consensus.cc:2454] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:43.483711 31899 leader_election.cc:302] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:43.484130 32432 raft_consensus.cc:2781] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:43.484262 32432 raft_consensus.cc:686] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:43.484567 32432 consensus_queue.cc:237] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.488606 32494 raft_consensus.cc:482] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.488883 32494 raft_consensus.cc:504] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:43.495720 32433 raft_consensus.cc:482] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.495934 32433 raft_consensus.cc:504] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.498428 32433 leader_election.cc:288] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:43.509122 29098 catalog_manager.cc:5484] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:43.511484 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:43.513957 32483 ts_tablet_manager.cc:1398] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.060s user 0.004s sys 0.011s
I20240501 14:00:43.514462 32483 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:43.516233 32483 tablet_bootstrap.cc:654] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.523173 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:00:43.526083 29442 leader_election.cc:341] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:43.526362 29442 leader_election.cc:302] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa; no voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:43.526693 32279 raft_consensus.cc:2726] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:43.529160 32494 leader_election.cc:288] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:43.534651 32483 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:43.534888 32483 ts_tablet_manager.cc:1367] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240501 14:00:43.536314 32483 raft_consensus.cc:348] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.536617 32483 raft_consensus.cc:374] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.536752 32483 raft_consensus.cc:729] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.537055 32483 consensus_queue.cc:260] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.537312 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:43.537537 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:43.537797 32483 ts_tablet_manager.cc:1398] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:43.538151 32483 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:43.540541 32483 tablet_bootstrap.cc:654] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.541534 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 0465b002ebf0448f9d45a3f07209059b
I20240501 14:00:43.543629 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b002ebf0448f9d45a3f07209059b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:43.543735 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b002ebf0448f9d45a3f07209059b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:43.544384 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b002ebf0448f9d45a3f07209059b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:43.546391 32483 tablet_bootstrap.cc:492] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:43.546561 32483 ts_tablet_manager.cc:1367] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:00:43.547291 32483 raft_consensus.cc:348] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.547539 32483 raft_consensus.cc:374] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.547623 32483 raft_consensus.cc:729] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.547860 32483 consensus_queue.cc:260] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.548338 32483 ts_tablet_manager.cc:1398] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:43.561359 32432 raft_consensus.cc:482] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.561605 32432 raft_consensus.cc:504] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.562590 32432 leader_election.cc:288] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:43.566061 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:43.566452 32097 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:43.566963 31900 leader_election.cc:302] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:43.567312 32432 raft_consensus.cc:2781] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:43.567464 32432 raft_consensus.cc:482] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:43.567574 32432 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.570616 32503 tablet_replica.cc:330] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:43.571141 32503 raft_consensus.cc:2229] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.571808 32503 raft_consensus.cc:2258] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.572621 32504 tablet_replica.cc:330] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:43.573535 32505 tablet_replica.cc:330] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:43.573529 32432 raft_consensus.cc:504] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.574898 32432 leader_election.cc:288] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:43.573944 32505 raft_consensus.cc:2229] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.575809 32505 raft_consensus.cc:2258] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.576161 32504 raft_consensus.cc:2229] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.576434 32504 raft_consensus.cc:2258] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.578060 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:43.578330 32097 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.579239 32504 ts_tablet_manager.cc:1875] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.583917 32097 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:43.584789 31900 leader_election.cc:302] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:43.585325 32432 raft_consensus.cc:2781] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:43.587078 32432 raft_consensus.cc:686] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:43.587610 32432 consensus_queue.cc:237] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:43.589841 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:43.591879 32503 ts_tablet_manager.cc:1875] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.595947 32505 ts_tablet_manager.cc:1875] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.598119 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:00:43.610183 32143 tablet.cc:2369] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:43.622306 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:43.622776 29245 raft_consensus.cc:2379] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a697f3a1ee544daa7073814ab845d58 in current term 1: Already voted for candidate 9903ed35bca144bcb74e9bf5102117e7 in this term.
I20240501 14:00:43.623886 32030 leader_election.cc:302] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:43.625150 32494 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.628233 32503 ts_tablet_manager.cc:1888] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.628404 32503 log.cc:1196] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/0465b002ebf0448f9d45a3f07209059b
I20240501 14:00:43.629484 32494 raft_consensus.cc:2726] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:43.640075 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:43.640473 32097 raft_consensus.cc:2379] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a446ac802a7457d9f27076781f21d79 in current term 1: Already voted for candidate 9903ed35bca144bcb74e9bf5102117e7 in this term.
I20240501 14:00:43.641767 32504 ts_tablet_manager.cc:1888] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.641952 32504 log.cc:1196] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/0465b002ebf0448f9d45a3f07209059b
I20240501 14:00:43.642541 32504 ts_tablet_manager.cc:1909] T 0465b002ebf0448f9d45a3f07209059b P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:43.644351 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table_new_name [id=8b528b4859b54372a6290061b12808c5]) successfully deleted
I20240501 14:00:43.641369 32163 leader_election.cc:302] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:43.645520 32433 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.616430 32505 ts_tablet_manager.cc:1888] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.646260 32503 ts_tablet_manager.cc:1909] T 0465b002ebf0448f9d45a3f07209059b P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:43.646404 32505 log.cc:1196] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/0465b002ebf0448f9d45a3f07209059b
I20240501 14:00:43.646967 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:43.649070 32433 raft_consensus.cc:2726] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:43.651579 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table_new_name [id=8b528b4859b54372a6290061b12808c5]) successfully deleted
I20240501 14:00:43.651718 29103 catalog_manager.cc:5484] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:43.653039 32505 ts_tablet_manager.cc:1909] T 0465b002ebf0448f9d45a3f07209059b P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:43.655002 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 0465b002ebf0448f9d45a3f07209059b (table rebalancer_dynamic_table_new_name [id=8b528b4859b54372a6290061b12808c5]) successfully deleted
I20240501 14:00:43.661747 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]), partition=
I20240501 14:00:43.662324 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da17f1b4d274544b43e7df5e38a0b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.664076 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]), partition=
I20240501 14:00:43.664265 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]), partition=
I20240501 14:00:43.664532 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da17f1b4d274544b43e7df5e38a0b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.664669 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da17f1b4d274544b43e7df5e38a0b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.669893 32426 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:43.671556 32426 tablet_bootstrap.cc:654] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.673830 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:43.674098 32227 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:43.673859 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:43.677865 32429 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:43.679404 32429 tablet_bootstrap.cc:654] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.677597 32228 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:43.688835 32429 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:43.689131 32426 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:43.689152 32429 ts_tablet_manager.cc:1367] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 14:00:43.689337 32426 ts_tablet_manager.cc:1367] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.008s
I20240501 14:00:43.690058 32429 raft_consensus.cc:348] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.690412 32429 raft_consensus.cc:374] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.690555 32429 raft_consensus.cc:729] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.690611 32426 raft_consensus.cc:348] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.691037 32426 raft_consensus.cc:374] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.690965 32429 consensus_queue.cc:260] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.691148 32426 raft_consensus.cc:729] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.689145 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:00:43.691491 32426 consensus_queue.cc:260] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.691610 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:00:43.692873 32426 ts_tablet_manager.cc:1398] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:43.693899 32429 ts_tablet_manager.cc:1398] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:43.698773 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet f911e40875f2453e902e4cf1816578a6
I20240501 14:00:43.699570 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet f911e40875f2453e902e4cf1816578a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.699724 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet f911e40875f2453e902e4cf1816578a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:43.699846 32459 tablet_replica.cc:330] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.700047 32460 tablet_replica.cc:330] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:43.700067 32459 raft_consensus.cc:2229] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.700255 32460 raft_consensus.cc:2229] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.700302 32459 raft_consensus.cc:2258] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.700552 32460 raft_consensus.cc:2258] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.700713 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 18da3f23b58c47468987acd6d5b04269
I20240501 14:00:43.701113 32459 ts_tablet_manager.cc:1875] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.701344 32460 ts_tablet_manager.cc:1875] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.702312 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 18da3f23b58c47468987acd6d5b04269 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:43.702600 32461 tablet_replica.cc:330] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:43.702803 32461 raft_consensus.cc:2229] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.703147 32461 raft_consensus.cc:2258] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.703239 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 18da3f23b58c47468987acd6d5b04269 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:43.703527 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 770a6a8f192c430aa00e921af46543ba
I20240501 14:00:43.703554 32462 tablet_replica.cc:330] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:43.703740 32462 raft_consensus.cc:2229] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.703867 32461 ts_tablet_manager.cc:1875] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.704010 32462 raft_consensus.cc:2258] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.704126 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 770a6a8f192c430aa00e921af46543ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:43.704706 32462 ts_tablet_manager.cc:1875] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.704810 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 770a6a8f192c430aa00e921af46543ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.705093 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet f911e40875f2453e902e4cf1816578a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:43.705670 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 18da3f23b58c47468987acd6d5b04269 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:43.706818 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 770a6a8f192c430aa00e921af46543ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:43.711130 32460 ts_tablet_manager.cc:1888] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.711334 32460 log.cc:1196] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/f911e40875f2453e902e4cf1816578a6
I20240501 14:00:43.711966 32460 ts_tablet_manager.cc:1909] T f911e40875f2453e902e4cf1816578a6 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:43.713424 32516 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:43.713682 32461 ts_tablet_manager.cc:1888] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.713832 32461 log.cc:1196] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/18da3f23b58c47468987acd6d5b04269
I20240501 14:00:43.714258 32461 ts_tablet_manager.cc:1909] T 18da3f23b58c47468987acd6d5b04269 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:43.715337 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 18da3f23b58c47468987acd6d5b04269 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.715482 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet f911e40875f2453e902e4cf1816578a6 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.715866 32460 tablet_replica.cc:330] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:43.716138 32460 raft_consensus.cc:2229] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.716403 32460 raft_consensus.cc:2258] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.716660 32459 ts_tablet_manager.cc:1888] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.716809 32459 log.cc:1196] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/f911e40875f2453e902e4cf1816578a6
I20240501 14:00:43.717274 32460 ts_tablet_manager.cc:1875] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.719966 32459 ts_tablet_manager.cc:1909] T f911e40875f2453e902e4cf1816578a6 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.720019 32516 tablet_bootstrap.cc:654] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.720176 32462 ts_tablet_manager.cc:1888] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.721087 32462 log.cc:1196] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/18da3f23b58c47468987acd6d5b04269
I20240501 14:00:43.723073 32459 tablet_replica.cc:330] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.723181 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet f911e40875f2453e902e4cf1816578a6 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
W20240501 14:00:43.723959 29570 consensus_peers.cc:485] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:43.724246 32462 ts_tablet_manager.cc:1909] T 18da3f23b58c47468987acd6d5b04269 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:43.724548 32459 raft_consensus.cc:2229] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.725761 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 18da3f23b58c47468987acd6d5b04269 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.726161 32459 raft_consensus.cc:2258] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.726795 32459 ts_tablet_manager.cc:1875] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.728484 32462 tablet_replica.cc:330] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:43.728761 32462 raft_consensus.cc:2229] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.729960 32462 raft_consensus.cc:2258] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.730733 32462 ts_tablet_manager.cc:1875] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.730926 32516 tablet_bootstrap.cc:492] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:43.731149 32516 ts_tablet_manager.cc:1367] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20240501 14:00:43.732039 32516 raft_consensus.cc:348] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.732319 32516 raft_consensus.cc:374] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.732458 32516 raft_consensus.cc:729] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.732805 32460 ts_tablet_manager.cc:1888] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.732959 32460 log.cc:1196] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/770a6a8f192c430aa00e921af46543ba
I20240501 14:00:43.732786 32516 consensus_queue.cc:260] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.733440 32460 ts_tablet_manager.cc:1909] T 770a6a8f192c430aa00e921af46543ba P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:43.734373 32516 ts_tablet_manager.cc:1398] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:43.734375 32459 ts_tablet_manager.cc:1888] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.734513 32459 log.cc:1196] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/770a6a8f192c430aa00e921af46543ba
I20240501 14:00:43.734583 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 770a6a8f192c430aa00e921af46543ba (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.734992 32459 ts_tablet_manager.cc:1909] T 770a6a8f192c430aa00e921af46543ba P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.736107 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 770a6a8f192c430aa00e921af46543ba (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.741118 32459 tablet_replica.cc:330] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:43.741421 32459 raft_consensus.cc:2229] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.741789 32459 raft_consensus.cc:2258] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.742491 32459 ts_tablet_manager.cc:1875] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.744921 32462 ts_tablet_manager.cc:1888] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.745136 32462 log.cc:1196] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/770a6a8f192c430aa00e921af46543ba
I20240501 14:00:43.745783 32462 ts_tablet_manager.cc:1909] T 770a6a8f192c430aa00e921af46543ba P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:43.747087 32462 tablet_replica.cc:330] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:43.747215 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 770a6a8f192c430aa00e921af46543ba (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.748163 32462 raft_consensus.cc:2229] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.748819 32459 ts_tablet_manager.cc:1888] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.749022 32459 log.cc:1196] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/18da3f23b58c47468987acd6d5b04269
I20240501 14:00:43.749511 32459 ts_tablet_manager.cc:1909] T 18da3f23b58c47468987acd6d5b04269 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:43.750531 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 18da3f23b58c47468987acd6d5b04269 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.753727 32462 raft_consensus.cc:2258] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.754716 32462 ts_tablet_manager.cc:1875] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.771624 32432 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.774009 32462 ts_tablet_manager.cc:1888] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:43.774223 32462 log.cc:1196] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/f911e40875f2453e902e4cf1816578a6
I20240501 14:00:43.774749 32462 ts_tablet_manager.cc:1909] T f911e40875f2453e902e4cf1816578a6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:43.776468 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet f911e40875f2453e902e4cf1816578a6 (table rebalance_test_table_2 [id=c64c19a0e53e470fa27d4624a701e6e8]) successfully deleted
I20240501 14:00:43.780174 32479 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.906926 32434 raft_consensus.cc:482] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.907099 32434 raft_consensus.cc:504] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.907881 32434 leader_election.cc:288] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:43.908360 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da17f1b4d274544b43e7df5e38a0b23" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:43.908434 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da17f1b4d274544b43e7df5e38a0b23" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:43.908607 31964 raft_consensus.cc:2454] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:43.908666 32228 raft_consensus.cc:2454] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:43.909257 32295 leader_election.cc:302] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:43.909619 32434 raft_consensus.cc:2781] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:43.909752 32434 raft_consensus.cc:482] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:43.909847 32434 raft_consensus.cc:3037] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.912297 32434 raft_consensus.cc:504] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.914162 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da17f1b4d274544b43e7df5e38a0b23" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:43.914402 32228 raft_consensus.cc:3037] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.918078 32228 raft_consensus.cc:2454] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:43.918645 32295 leader_election.cc:302] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:43.921473 32573 raft_consensus.cc:2781] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:43.922222 32434 leader_election.cc:288] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:43.924747 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da17f1b4d274544b43e7df5e38a0b23" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:43.927227 31964 raft_consensus.cc:3037] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.937316 31964 raft_consensus.cc:2454] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:43.947666 32573 raft_consensus.cc:686] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:43.948067 32573 consensus_queue.cc:237] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.957057 29103 catalog_manager.cc:5484] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:43.962105 32494 raft_consensus.cc:482] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.962316 32494 raft_consensus.cc:504] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:43.963177 32494 leader_election.cc:288] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:43.964299 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:43.997491 32097 raft_consensus.cc:3037] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.001899 32479 consensus_queue.cc:1035] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.028632 32432 consensus_queue.cc:1035] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.040823 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f354a7a3c4f4c61a62f2c3e0cbbf812" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:44.041477 32031 leader_election.cc:302] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:44.041875 32494 raft_consensus.cc:2726] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:44.079838 32075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:44.087164 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:44.095993 29224 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:00:44.103446 32432 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.120101 32432 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.153323 29487 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:44.169615 29357 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:00:44.171501 31945 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:44.172005 32228 raft_consensus.cc:1261] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:44.172549 32573 consensus_queue.cc:1035] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.181329 29615 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:44.186025 31964 raft_consensus.cc:1261] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:44.201481 32573 consensus_queue.cc:1035] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240501 14:00:44.207554 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:00:44.207811 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:00:44.212132 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet d09ea61ef29d4e1bb1a20d866b1e8051
I20240501 14:00:44.213297 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:44.213657 32460 tablet_replica.cc:330] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:44.213819 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:44.213868 32460 raft_consensus.cc:2229] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.214067 32461 tablet_replica.cc:330] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:44.214293 32461 raft_consensus.cc:2229] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.214418 32460 raft_consensus.cc:2258] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.214519 32461 raft_consensus.cc:2258] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.215191 32460 ts_tablet_manager.cc:1875] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.215246 32461 ts_tablet_manager.cc:1875] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.218706 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet d7cebf8b40474bf2b500e15fbf3c669c
I20240501 14:00:44.218959 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet d09ea61ef29d4e1bb1a20d866b1e8051 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.219264 32459 tablet_replica.cc:330] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.219480 32459 raft_consensus.cc:2229] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.219693 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:44.219758 32459 raft_consensus.cc:2258] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.220327 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:44.220474 32459 ts_tablet_manager.cc:1875] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.220620 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cebf8b40474bf2b500e15fbf3c669c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.223066 32462 tablet_replica.cc:330] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:44.223284 32462 raft_consensus.cc:2229] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.223501 32462 raft_consensus.cc:2258] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.223551 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 5be28b16ee2a4594b53a99ca80367596
I20240501 14:00:44.224185 32462 ts_tablet_manager.cc:1875] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.224236 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be28b16ee2a4594b53a99ca80367596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:44.225275 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be28b16ee2a4594b53a99ca80367596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:44.228143 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be28b16ee2a4594b53a99ca80367596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.230706 32461 ts_tablet_manager.cc:1888] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.231751 32461 log.cc:1196] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/d09ea61ef29d4e1bb1a20d866b1e8051
I20240501 14:00:44.239830 32462 ts_tablet_manager.cc:1888] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.240008 32462 log.cc:1196] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/d7cebf8b40474bf2b500e15fbf3c669c
I20240501 14:00:44.240192 32461 ts_tablet_manager.cc:1909] T d09ea61ef29d4e1bb1a20d866b1e8051 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:44.240432 32462 ts_tablet_manager.cc:1909] T d7cebf8b40474bf2b500e15fbf3c669c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:44.241379 32462 tablet_replica.cc:330] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:44.241469 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet d7cebf8b40474bf2b500e15fbf3c669c (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.241719 32462 raft_consensus.cc:2229] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.242028 32462 raft_consensus.cc:2258] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.242702 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet d09ea61ef29d4e1bb1a20d866b1e8051 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.243091 32461 tablet_replica.cc:330] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:44.243450 32461 raft_consensus.cc:2229] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.243830 32461 raft_consensus.cc:2258] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.245645 32462 ts_tablet_manager.cc:1875] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.246096 32461 ts_tablet_manager.cc:1875] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.247191 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:44.258436 29103 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]). Expected version 1 got 0
I20240501 14:00:44.259938 32442 mvcc.cc:205] Tried to move back new op lower bound from 7022887092917895168 to 7022887092015050752. Current Snapshot: MvccSnapshot[applied={T|T < 7022887092917895168}]
I20240501 14:00:44.264550 32075 tablet_service.cc:1467] Processing CreateTablet for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.266242 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 820be6be18af4781ace626f239379cff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.266635 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820be6be18af4781ace626f239379cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.244906 32459 ts_tablet_manager.cc:1888] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.267455 32459 log.cc:1196] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/d09ea61ef29d4e1bb1a20d866b1e8051
I20240501 14:00:44.267930 32459 ts_tablet_manager.cc:1909] T d09ea61ef29d4e1bb1a20d866b1e8051 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.268879 32460 ts_tablet_manager.cc:1888] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.269071 32460 log.cc:1196] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/d09ea61ef29d4e1bb1a20d866b1e8051
I20240501 14:00:44.271798 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.273062 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.273738 32460 ts_tablet_manager.cc:1909] T d09ea61ef29d4e1bb1a20d866b1e8051 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:44.275002 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.275439 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.276454 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet d09ea61ef29d4e1bb1a20d866b1e8051 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.276844 32459 tablet_replica.cc:330] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.277109 32459 raft_consensus.cc:2229] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.277437 32459 raft_consensus.cc:2258] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.277415 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 820be6be18af4781ace626f239379cff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.277843 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820be6be18af4781ace626f239379cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.278403 32604 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.278930 32604 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:44.283057 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.285427 32460 tablet_replica.cc:330] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:44.286028 32341 tablet_service.cc:1467] Processing CreateTablet for tablet e5e2a1f3418c477aa65b77bfa171a5a1 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.286474 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e2a1f3418c477aa65b77bfa171a5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.287247 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet d09ea61ef29d4e1bb1a20d866b1e8051 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.289412 29615 tablet_service.cc:1467] Processing CreateTablet for tablet e5e2a1f3418c477aa65b77bfa171a5a1 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.289618 32459 ts_tablet_manager.cc:1875] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.289839 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e2a1f3418c477aa65b77bfa171a5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.292637 29103 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]). Expected version 1 got 0
I20240501 14:00:44.292027 32076 tablet_service.cc:1467] Processing CreateTablet for tablet e5e2a1f3418c477aa65b77bfa171a5a1 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.293031 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e2a1f3418c477aa65b77bfa171a5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.293392 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 820be6be18af4781ace626f239379cff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.293776 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820be6be18af4781ace626f239379cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.303107 32462 ts_tablet_manager.cc:1888] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.303256 32462 log.cc:1196] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/5be28b16ee2a4594b53a99ca80367596
I20240501 14:00:44.303738 32462 ts_tablet_manager.cc:1909] T 5be28b16ee2a4594b53a99ca80367596 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:44.311597 32208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:44.314000 32609 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:44.314102 29098 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]). Expected version 1 got 0
I20240501 14:00:44.316315 32609 tablet_bootstrap.cc:654] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.319720 32608 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:44.321421 32606 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:44.323474 32606 tablet_bootstrap.cc:654] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.325984 32607 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:44.326741 32460 raft_consensus.cc:2229] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.326854 32609 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:44.327010 32609 ts_tablet_manager.cc:1367] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 14:00:44.327190 32460 raft_consensus.cc:2258] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.327762 32607 tablet_bootstrap.cc:654] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.327888 32609 raft_consensus.cc:348] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.328121 32609 raft_consensus.cc:374] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.328228 32609 raft_consensus.cc:729] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.328415 32460 ts_tablet_manager.cc:1875] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.328491 32609 consensus_queue.cc:260] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.329262 32609 ts_tablet_manager.cc:1398] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:44.329569 32609 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:44.330454 32610 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.332698 29098 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]). Expected version 1 got 0
I20240501 14:00:44.333264 32609 tablet_bootstrap.cc:654] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.334028 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 5be28b16ee2a4594b53a99ca80367596 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.332523 32603 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.335136 32608 tablet_bootstrap.cc:654] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.335685 32603 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:44.336347 32610 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:44.342840 32606 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:44.343814 32608 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:44.344892 32607 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:44.345528 32608 ts_tablet_manager.cc:1367] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 14:00:44.346975 32606 ts_tablet_manager.cc:1367] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 14:00:44.347472 32607 ts_tablet_manager.cc:1367] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 14:00:44.348451 32608 raft_consensus.cc:348] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.349501 32606 raft_consensus.cc:348] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.350602 32607 raft_consensus.cc:348] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.351727 32606 raft_consensus.cc:374] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.351873 32608 raft_consensus.cc:374] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.352099 32606 raft_consensus.cc:729] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.352248 32607 raft_consensus.cc:374] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.352506 32608 raft_consensus.cc:729] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.352813 32607 raft_consensus.cc:729] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.353142 32606 consensus_queue.cc:260] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.353601 32608 consensus_queue.cc:260] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.354133 32607 consensus_queue.cc:260] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.356623 32606 ts_tablet_manager.cc:1398] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:44.357100 32608 ts_tablet_manager.cc:1398] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:44.357744 32607 ts_tablet_manager.cc:1398] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:44.359449 32433 raft_consensus.cc:482] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.359666 32433 raft_consensus.cc:504] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.360621 32433 leader_election.cc:288] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:44.354279 29098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce] alter complete (version 1)
I20240501 14:00:44.365195 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820be6be18af4781ace626f239379cff" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:00:44.366174 32163 leader_election.cc:341] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:44.384416 29683 tablet.cc:2369] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:44.384366 32606 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:44.390681 32606 tablet_bootstrap.cc:654] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.397780 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820be6be18af4781ace626f239379cff" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:44.398054 29246 raft_consensus.cc:2454] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:44.398571 32161 leader_election.cc:302] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:44.399127 32616 raft_consensus.cc:2781] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.399222 32616 raft_consensus.cc:482] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.399340 32616 raft_consensus.cc:3037] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.403244 32616 raft_consensus.cc:504] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.404367 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820be6be18af4781ace626f239379cff" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:44.404601 29246 raft_consensus.cc:3037] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.408078 29246 raft_consensus.cc:2454] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:44.408690 32161 leader_election.cc:302] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:44.409155 32433 raft_consensus.cc:2781] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.410661 32461 ts_tablet_manager.cc:1888] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.410907 32461 log.cc:1196] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/5be28b16ee2a4594b53a99ca80367596
I20240501 14:00:44.412324 32461 ts_tablet_manager.cc:1909] T 5be28b16ee2a4594b53a99ca80367596 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:44.413885 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 5be28b16ee2a4594b53a99ca80367596 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.427464 32459 ts_tablet_manager.cc:1888] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.432133 32341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:44.442510 32459 log.cc:1196] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7cebf8b40474bf2b500e15fbf3c669c
I20240501 14:00:44.444020 32459 ts_tablet_manager.cc:1909] T d7cebf8b40474bf2b500e15fbf3c669c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.452755 32433 raft_consensus.cc:686] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:44.453404 32433 consensus_queue.cc:237] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.455211 32459 tablet_replica.cc:330] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.455324 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet d7cebf8b40474bf2b500e15fbf3c669c (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.455385 32605 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:44.459561 32459 raft_consensus.cc:2229] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.459893 32459 raft_consensus.cc:2258] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.461141 32459 ts_tablet_manager.cc:1875] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.464761 32460 ts_tablet_manager.cc:1888] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.465721 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820be6be18af4781ace626f239379cff" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
W20240501 14:00:44.466410 32163 leader_election.cc:341] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:44.466676 32616 leader_election.cc:288] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:44.466971 32015 raft_consensus.cc:482] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.467146 32015 raft_consensus.cc:504] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.468142 32015 leader_election.cc:288] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:44.472039 29103 catalog_manager.cc:5484] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:44.476487 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:44.477665 32605 tablet_bootstrap.cc:654] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.478132 32609 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:44.478315 32609 ts_tablet_manager.cc:1367] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.149s user 0.036s sys 0.070s
I20240501 14:00:44.479151 32609 raft_consensus.cc:348] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.479379 32609 raft_consensus.cc:374] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.479446 32609 raft_consensus.cc:729] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.479725 32609 consensus_queue.cc:260] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.480319 32609 ts_tablet_manager.cc:1398] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:44.480635 32609 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:44.482286 32609 tablet_bootstrap.cc:654] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.487864 32609 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:44.488018 32609 ts_tablet_manager.cc:1367] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:44.488147 32604 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.488776 32604 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:44.488763 32609 raft_consensus.cc:348] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.488946 32609 raft_consensus.cc:374] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.489065 32609 raft_consensus.cc:729] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.489326 32609 consensus_queue.cc:260] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.489953 32609 ts_tablet_manager.cc:1398] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:44.495558 32610 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.496152 32610 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:44.501822 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=86b77cf50a3e43f29a8eb01bf5d5c8ce] alter complete (version 2)
I20240501 14:00:44.508497 32606 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:44.512512 32606 ts_tablet_manager.cc:1367] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.128s user 0.004s sys 0.000s
I20240501 14:00:44.514860 32603 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.515466 32603 tablet.cc:1620] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:44.516466 32606 raft_consensus.cc:348] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.517706 32606 raft_consensus.cc:374] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.517982 32606 raft_consensus.cc:729] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.518775 32606 consensus_queue.cc:260] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.515341 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:00:44.522109 29571 leader_election.cc:341] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:44.523069 32606 ts_tablet_manager.cc:1398] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:44.523348 32460 log.cc:1196] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7cebf8b40474bf2b500e15fbf3c669c
I20240501 14:00:44.523880 32460 ts_tablet_manager.cc:1909] T d7cebf8b40474bf2b500e15fbf3c669c P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:44.527102 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet d7cebf8b40474bf2b500e15fbf3c669c (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.529361 32459 ts_tablet_manager.cc:1888] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.529556 32459 log.cc:1196] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/5be28b16ee2a4594b53a99ca80367596
I20240501 14:00:44.530680 32459 ts_tablet_manager.cc:1909] T 5be28b16ee2a4594b53a99ca80367596 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.532151 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 5be28b16ee2a4594b53a99ca80367596 (table rebalance_test_table_4 [id=e14556c6568e4077a3d98898c6655bd7]) successfully deleted
I20240501 14:00:44.532521 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:44.532765 32096 raft_consensus.cc:2454] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:44.533285 29572 leader_election.cc:302] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:44.533666 32015 raft_consensus.cc:2781] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.533804 32015 raft_consensus.cc:482] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.533914 32015 raft_consensus.cc:3037] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.538096 32015 raft_consensus.cc:504] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.539408 32015 leader_election.cc:288] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:44.539654 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:44.540316 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:44.540529 32096 raft_consensus.cc:3037] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.540938 32605 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:44.541103 32605 ts_tablet_manager.cc:1367] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20240501 14:00:44.542115 32605 raft_consensus.cc:348] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.542308 32605 raft_consensus.cc:374] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.542399 32605 raft_consensus.cc:729] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.542655 32605 consensus_queue.cc:260] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.543449 32096 raft_consensus.cc:2454] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:44.544032 32605 ts_tablet_manager.cc:1398] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 14:00:44.544102 29571 leader_election.cc:341] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:44.544441 32605 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:44.544709 29572 leader_election.cc:302] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:44.545087 32015 raft_consensus.cc:2781] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.545665 32015 raft_consensus.cc:686] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:44.546030 32015 consensus_queue.cc:237] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.547369 32605 tablet_bootstrap.cc:654] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.563630 29098 catalog_manager.cc:5484] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:44.571611 32605 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:44.571830 32605 ts_tablet_manager.cc:1367] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240501 14:00:44.572791 32605 raft_consensus.cc:348] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.573149 32605 raft_consensus.cc:374] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.573267 32605 raft_consensus.cc:729] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.573557 32605 consensus_queue.cc:260] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.574223 32605 ts_tablet_manager.cc:1398] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:44.603826 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:44.621739 32603 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.632061 32604 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.641219 32610 tablet.cc:1722] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.662801 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=86b77cf50a3e43f29a8eb01bf5d5c8ce] alter complete (version 3)
I20240501 14:00:44.674893 32494 raft_consensus.cc:482] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.675079 32494 raft_consensus.cc:504] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.675848 32494 leader_election.cc:288] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:44.693946 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bdc21d1dc644e7ac44f5b9ad39ab32" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:44.694262 32362 raft_consensus.cc:2454] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:44.697213 32031 leader_election.cc:302] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:44.697564 32494 raft_consensus.cc:2781] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.697669 32494 raft_consensus.cc:482] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.697780 32494 raft_consensus.cc:3037] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.700567 32494 raft_consensus.cc:504] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.701437 32494 leader_election.cc:288] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:44.702369 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bdc21d1dc644e7ac44f5b9ad39ab32" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:44.702394 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bdc21d1dc644e7ac44f5b9ad39ab32" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:44.702589 32362 raft_consensus.cc:3037] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.702718 32227 raft_consensus.cc:2454] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:44.705731 32362 raft_consensus.cc:2454] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:44.706645 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bdc21d1dc644e7ac44f5b9ad39ab32" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:44.706859 32227 raft_consensus.cc:3037] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.709554 32227 raft_consensus.cc:2454] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:44.710057 32031 leader_election.cc:302] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:44.710470 32494 raft_consensus.cc:2781] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.711074 32494 raft_consensus.cc:686] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:44.711622 32494 consensus_queue.cc:237] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:44.715167 29098 catalog_manager.cc:5484] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:44.732339 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:00:44.732612 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:00:44.736460 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet da05f71e097a4661b687e94d82b2133a
I20240501 14:00:44.737463 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet da05f71e097a4661b687e94d82b2133a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.737820 32459 tablet_replica.cc:330] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.737900 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet da05f71e097a4661b687e94d82b2133a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:44.738114 32459 raft_consensus.cc:2229] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.738389 32459 raft_consensus.cc:2258] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.738421 32462 tablet_replica.cc:330] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:44.738673 32462 raft_consensus.cc:2229] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.738940 32462 raft_consensus.cc:2258] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.739323 32459 ts_tablet_manager.cc:1875] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.739765 32462 ts_tablet_manager.cc:1875] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.741976 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet da05f71e097a4661b687e94d82b2133a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:44.742287 32460 tablet_replica.cc:330] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:44.742529 32460 raft_consensus.cc:2229] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.742941 32460 raft_consensus.cc:2258] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.743741 32460 ts_tablet_manager.cc:1875] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.745944 32459 ts_tablet_manager.cc:1888] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.746090 32459 log.cc:1196] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/da05f71e097a4661b687e94d82b2133a
I20240501 14:00:44.746630 32459 ts_tablet_manager.cc:1909] T da05f71e097a4661b687e94d82b2133a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.747700 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet da05f71e097a4661b687e94d82b2133a (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.748822 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 9173d6ec3fe44271bcec61a0b32d5c4d
I20240501 14:00:44.749732 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:44.750236 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:44.750530 32461 tablet_replica.cc:330] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:44.750720 32460 ts_tablet_manager.cc:1888] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.750761 32461 raft_consensus.cc:2229] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.750839 32460 log.cc:1196] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/da05f71e097a4661b687e94d82b2133a
I20240501 14:00:44.751015 32461 raft_consensus.cc:2258] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.751273 32460 ts_tablet_manager.cc:1909] T da05f71e097a4661b687e94d82b2133a P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:44.751777 32461 ts_tablet_manager.cc:1875] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.756296 32461 ts_tablet_manager.cc:1888] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.756457 32461 log.cc:1196] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/9173d6ec3fe44271bcec61a0b32d5c4d
I20240501 14:00:44.756464 32462 ts_tablet_manager.cc:1888] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.756947 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet da05f71e097a4661b687e94d82b2133a (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.757439 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 58828a3d248c4e57908eeaaf9f880adc
I20240501 14:00:44.758473 32462 log.cc:1196] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/da05f71e097a4661b687e94d82b2133a
I20240501 14:00:44.758545 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 58828a3d248c4e57908eeaaf9f880adc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:44.759055 32462 ts_tablet_manager.cc:1909] T da05f71e097a4661b687e94d82b2133a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:44.759696 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 9173d6ec3fe44271bcec61a0b32d5c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.759796 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 58828a3d248c4e57908eeaaf9f880adc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:44.759991 32459 tablet_replica.cc:330] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.760088 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 58828a3d248c4e57908eeaaf9f880adc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:44.760195 32459 raft_consensus.cc:2229] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.760545 32459 raft_consensus.cc:2258] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.760674 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet da05f71e097a4661b687e94d82b2133a (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.761056 32462 tablet_replica.cc:330] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:44.761309 32462 raft_consensus.cc:2229] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.761401 32459 ts_tablet_manager.cc:1875] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.761551 32462 raft_consensus.cc:2258] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.762256 32462 ts_tablet_manager.cc:1875] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.763257 32461 ts_tablet_manager.cc:1909] T 9173d6ec3fe44271bcec61a0b32d5c4d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:44.765663 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 9173d6ec3fe44271bcec61a0b32d5c4d (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.767585 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:44.767859 32459 ts_tablet_manager.cc:1888] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.767827 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:44.767971 32459 log.cc:1196] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/9173d6ec3fe44271bcec61a0b32d5c4d
I20240501 14:00:44.768432 32459 ts_tablet_manager.cc:1909] T 9173d6ec3fe44271bcec61a0b32d5c4d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.768352 32460 tablet_replica.cc:330] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:44.768913 32460 raft_consensus.cc:2229] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.769726 32460 raft_consensus.cc:2258] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.770978 32460 ts_tablet_manager.cc:1875] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.771581 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 0da17f1b4d274544b43e7df5e38a0b23
I20240501 14:00:44.773322 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:44.773661 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:44.774120 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da17f1b4d274544b43e7df5e38a0b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:44.774223 32462 ts_tablet_manager.cc:1888] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.774369 32462 log.cc:1196] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/9173d6ec3fe44271bcec61a0b32d5c4d
I20240501 14:00:44.774909 32462 ts_tablet_manager.cc:1909] T 9173d6ec3fe44271bcec61a0b32d5c4d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:44.776163 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 9173d6ec3fe44271bcec61a0b32d5c4d (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.776551 32462 tablet_replica.cc:330] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:44.776958 32462 raft_consensus.cc:2229] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.777244 32462 raft_consensus.cc:2258] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.777293 32459 tablet_replica.cc:330] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:44.777465 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 9173d6ec3fe44271bcec61a0b32d5c4d (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.777537 32459 raft_consensus.cc:2229] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.777781 32459 raft_consensus.cc:2258] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.778167 32462 ts_tablet_manager.cc:1875] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.779268 32459 ts_tablet_manager.cc:1875] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.784732 32462 ts_tablet_manager.cc:1888] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.785001 32462 log.cc:1196] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/58828a3d248c4e57908eeaaf9f880adc
I20240501 14:00:44.785483 32462 ts_tablet_manager.cc:1909] T 58828a3d248c4e57908eeaaf9f880adc P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:44.786129 32459 ts_tablet_manager.cc:1888] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.786278 32459 log.cc:1196] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/58828a3d248c4e57908eeaaf9f880adc
I20240501 14:00:44.786695 32459 ts_tablet_manager.cc:1909] T 58828a3d248c4e57908eeaaf9f880adc P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:44.786741 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 58828a3d248c4e57908eeaaf9f880adc (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.788101 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 58828a3d248c4e57908eeaaf9f880adc (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.789431 32649 tablet_replica.cc:330] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:44.789991 32648 tablet_replica.cc:330] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:44.790794 32648 raft_consensus.cc:2229] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.791146 32649 raft_consensus.cc:2229] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.791465 32648 raft_consensus.cc:2258] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.793130 32649 raft_consensus.cc:2258] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.797614 32648 ts_tablet_manager.cc:1875] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.803607 32649 ts_tablet_manager.cc:1875] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.809463 32648 ts_tablet_manager.cc:1888] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.809765 32648 log.cc:1196] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/0da17f1b4d274544b43e7df5e38a0b23
I20240501 14:00:44.810386 32648 ts_tablet_manager.cc:1909] T 0da17f1b4d274544b43e7df5e38a0b23 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:44.811427 32647 tablet_replica.cc:330] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:44.811722 32647 raft_consensus.cc:2229] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.811744 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table_new_name [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]) successfully deleted
I20240501 14:00:44.811982 32647 raft_consensus.cc:2258] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.813390 32647 ts_tablet_manager.cc:1875] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.816681 32460 ts_tablet_manager.cc:1888] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:44.816884 32460 log.cc:1196] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/58828a3d248c4e57908eeaaf9f880adc
I20240501 14:00:44.820222 32647 ts_tablet_manager.cc:1888] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.820436 32647 log.cc:1196] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/0da17f1b4d274544b43e7df5e38a0b23
I20240501 14:00:44.821014 32647 ts_tablet_manager.cc:1909] T 0da17f1b4d274544b43e7df5e38a0b23 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:44.821949 32460 ts_tablet_manager.cc:1909] T 58828a3d248c4e57908eeaaf9f880adc P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:44.822161 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table_new_name [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]) successfully deleted
I20240501 14:00:44.825074 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 58828a3d248c4e57908eeaaf9f880adc (table rebalance_test_table_6 [id=87afc2017a564388adb8e0aa9d0a129c]) successfully deleted
I20240501 14:00:44.826630 32649 ts_tablet_manager.cc:1888] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.826781 32649 log.cc:1196] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/0da17f1b4d274544b43e7df5e38a0b23
I20240501 14:00:44.827303 32649 ts_tablet_manager.cc:1909] T 0da17f1b4d274544b43e7df5e38a0b23 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:44.829355 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 0da17f1b4d274544b43e7df5e38a0b23 (table rebalancer_dynamic_table_new_name [id=86b77cf50a3e43f29a8eb01bf5d5c8ce]) successfully deleted
I20240501 14:00:44.875203 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:44.881723 32096 raft_consensus.cc:3037] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.884655 31945 tablet_service.cc:1467] Processing CreateTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]), partition=
I20240501 14:00:44.884779 32075 tablet_service.cc:1467] Processing CreateTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]), partition=
I20240501 14:00:44.885102 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b69db836da4f3193717e0b8c9f5ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.885177 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b69db836da4f3193717e0b8c9f5ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.886428 29226 tablet_service.cc:1467] Processing CreateTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]), partition=
I20240501 14:00:44.886775 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b69db836da4f3193717e0b8c9f5ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.889396 32433 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.894940 32609 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:44.897186 32609 tablet_bootstrap.cc:654] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.911048 32433 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.917899 32660 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:44.923192 32609 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:44.923398 32609 ts_tablet_manager.cc:1367] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.002s
I20240501 14:00:44.924199 32609 raft_consensus.cc:348] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.931097 32609 raft_consensus.cc:374] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.931195 32609 raft_consensus.cc:729] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.931483 32609 consensus_queue.cc:260] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.932013 32609 ts_tablet_manager.cc:1398] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:00:44.933367 32660 tablet_bootstrap.cc:654] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.947146 32660 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:44.947336 32660 ts_tablet_manager.cc:1367] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 14:00:44.948182 32660 raft_consensus.cc:348] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.948428 32660 raft_consensus.cc:374] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.948525 32660 raft_consensus.cc:729] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.951292 32574 raft_consensus.cc:482] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.951464 32574 raft_consensus.cc:504] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:44.952451 32574 leader_election.cc:288] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:44.937583 32658 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:44.955495 32658 tablet_bootstrap.cc:654] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.956395 32660 consensus_queue.cc:260] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.956995 32660 ts_tablet_manager.cc:1398] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:00:44.959470 32658 tablet_bootstrap.cc:492] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:44.959619 32658 ts_tablet_manager.cc:1367] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:00:44.960409 32658 raft_consensus.cc:348] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.960772 32658 raft_consensus.cc:374] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.960954 32658 raft_consensus.cc:729] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.961277 32658 consensus_queue.cc:260] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:44.962565 32658 ts_tablet_manager.cc:1398] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:00:44.977989 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:44.985939 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e2a1f3418c477aa65b77bfa171a5a1" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:44.986240 32097 raft_consensus.cc:2379] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6edd6a3430d44c38a19c2804e555059f in current term 1: Already voted for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in this term.
I20240501 14:00:44.987296 32296 leader_election.cc:302] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:44.987656 32574 raft_consensus.cc:3037] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.991011 32574 raft_consensus.cc:2726] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:45.024920 32015 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.033185 32479 raft_consensus.cc:482] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.033382 32479 raft_consensus.cc:504] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.034149 32479 leader_election.cc:288] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:45.035131 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b69db836da4f3193717e0b8c9f5ab3" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:45.035394 32097 raft_consensus.cc:2454] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:45.036507 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b69db836da4f3193717e0b8c9f5ab3" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:45.036742 29246 raft_consensus.cc:2454] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:45.037477 31900 leader_election.cc:302] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.041061 32479 raft_consensus.cc:2781] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.041191 32479 raft_consensus.cc:482] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.041270 32479 raft_consensus.cc:3037] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.044138 32479 raft_consensus.cc:504] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.045883 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b69db836da4f3193717e0b8c9f5ab3" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:45.046121 29246 raft_consensus.cc:3037] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.046109 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b69db836da4f3193717e0b8c9f5ab3" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:45.046330 32097 raft_consensus.cc:3037] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.049685 29246 raft_consensus.cc:2454] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:45.050981 32097 raft_consensus.cc:2454] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:45.051504 31900 leader_election.cc:302] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.052987 32479 leader_election.cc:288] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:45.053225 32479 raft_consensus.cc:2781] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.053344 32479 raft_consensus.cc:686] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:45.053637 32479 consensus_queue.cc:237] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.098322 32494 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.098803 32693 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.113343 32641 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.125308 29100 catalog_manager.cc:5484] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:45.152158 32075 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:45.201871 29226 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:45.221469 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:45.231031 29487 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:45.235932 29226 tablet_service.cc:1467] Processing CreateTablet for tablet c8020272a8b14dea8a6a82802ea4e10a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.236487 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8020272a8b14dea8a6a82802ea4e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.236538 31944 tablet_service.cc:1467] Processing CreateTablet for tablet c8020272a8b14dea8a6a82802ea4e10a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.237053 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8020272a8b14dea8a6a82802ea4e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.237396 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.238286 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.238710 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.237466 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 36b0222ebb6d4246ab8551d5579b1760 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.241657 32075 tablet_service.cc:1467] Processing CreateTablet for tablet 36b0222ebb6d4246ab8551d5579b1760 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.242127 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b0222ebb6d4246ab8551d5579b1760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.242208 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 36b0222ebb6d4246ab8551d5579b1760 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.242622 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b0222ebb6d4246ab8551d5579b1760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.245293 32660 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:45.245352 32341 tablet_service.cc:1467] Processing CreateTablet for tablet c8020272a8b14dea8a6a82802ea4e10a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.245750 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8020272a8b14dea8a6a82802ea4e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.237841 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.251246 32660 tablet_bootstrap.cc:654] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.252494 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b0222ebb6d4246ab8551d5579b1760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.256281 32609 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:45.257891 32609 tablet_bootstrap.cc:654] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.261161 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.261679 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdebc6e2fdd4b2daa399cd1301fcde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.262526 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:00:45.262763 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:00:45.265116 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:45.267586 29356 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:45.274737 29615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:45.277379 32658 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:45.279354 32658 tablet_bootstrap.cc:654] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.284317 32660 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:45.284525 32660 ts_tablet_manager.cc:1367] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
I20240501 14:00:45.285442 32660 raft_consensus.cc:348] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.285677 32660 raft_consensus.cc:374] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.285763 32660 raft_consensus.cc:729] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.286104 32660 consensus_queue.cc:260] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.287796 32660 ts_tablet_manager.cc:1398] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:45.288166 32660 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:45.290593 32660 tablet_bootstrap.cc:654] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.296159 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 426bc72f14fb4490b329ab52bf7ca69c
I20240501 14:00:45.297072 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:45.297535 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.298066 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 426bc72f14fb4490b329ab52bf7ca69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:45.299243 32460 tablet_replica.cc:330] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:45.299515 32460 raft_consensus.cc:2229] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.299786 32460 raft_consensus.cc:2258] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.300547 32460 ts_tablet_manager.cc:1875] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.300549 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet b4ad2362aa0341d1b4faf9907d143458
I20240501 14:00:45.301234 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad2362aa0341d1b4faf9907d143458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:45.301766 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad2362aa0341d1b4faf9907d143458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.302254 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad2362aa0341d1b4faf9907d143458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:45.303402 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet b508eaf4280143df96ef3c72d54194ee
I20240501 14:00:45.308915 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet b508eaf4280143df96ef3c72d54194ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:45.316434 32724 tablet_replica.cc:330] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:45.298329 32692 raft_consensus.cc:482] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.316860 32692 raft_consensus.cc:504] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.317734 32692 leader_election.cc:288] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:45.319046 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8020272a8b14dea8a6a82802ea4e10a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
W20240501 14:00:45.319762 31898 leader_election.cc:341] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:45.323596 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet b508eaf4280143df96ef3c72d54194ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.324229 32718 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:45.296836 32660 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:45.327067 32718 tablet_bootstrap.cc:654] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.322525 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet b508eaf4280143df96ef3c72d54194ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:45.329363 32660 ts_tablet_manager.cc:1367] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.005s
I20240501 14:00:45.329397 32460 ts_tablet_manager.cc:1888] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.329686 32724 raft_consensus.cc:2229] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.330993 31943 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:45.331075 32660 raft_consensus.cc:348] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.331883 32724 raft_consensus.cc:2258] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.333609 32460 log.cc:1196] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/426bc72f14fb4490b329ab52bf7ca69c
I20240501 14:00:45.334774 32460 ts_tablet_manager.cc:1909] T 426bc72f14fb4490b329ab52bf7ca69c P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:45.334825 32660 raft_consensus.cc:374] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.334949 32660 raft_consensus.cc:729] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.335201 32660 consensus_queue.cc:260] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.336390 32342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:45.336532 32724 ts_tablet_manager.cc:1875] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.342222 32609 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:45.349964 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 426bc72f14fb4490b329ab52bf7ca69c (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.350773 32609 ts_tablet_manager.cc:1367] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.004s
I20240501 14:00:45.351784 32609 raft_consensus.cc:348] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.352245 32460 tablet_replica.cc:330] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:45.352597 32460 raft_consensus.cc:2229] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.353020 32460 raft_consensus.cc:2258] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.353184 32609 raft_consensus.cc:374] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.354233 32460 ts_tablet_manager.cc:1875] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.354413 32609 raft_consensus.cc:729] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.354904 32609 consensus_queue.cc:260] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.356141 32609 ts_tablet_manager.cc:1398] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:45.366380 32658 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:45.366674 32658 ts_tablet_manager.cc:1367] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.009s
I20240501 14:00:45.367573 32658 raft_consensus.cc:348] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.368292 32658 raft_consensus.cc:374] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.368649 32658 raft_consensus.cc:729] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.369583 32658 consensus_queue.cc:260] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.371733 32658 ts_tablet_manager.cc:1398] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:00:45.372207 32660 ts_tablet_manager.cc:1398] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:00:45.372836 32660 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:45.374783 32660 tablet_bootstrap.cc:654] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.375461 32724 ts_tablet_manager.cc:1888] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.376605 32724 log.cc:1196] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/426bc72f14fb4490b329ab52bf7ca69c
I20240501 14:00:45.377036 32718 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:45.377202 32718 ts_tablet_manager.cc:1367] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.000s
I20240501 14:00:45.378077 32724 ts_tablet_manager.cc:1909] T 426bc72f14fb4490b329ab52bf7ca69c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:45.378934 32718 raft_consensus.cc:348] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.379206 32718 raft_consensus.cc:374] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.379355 32718 raft_consensus.cc:729] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.379688 32718 consensus_queue.cc:260] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.380641 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 426bc72f14fb4490b329ab52bf7ca69c (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.380860 32718 ts_tablet_manager.cc:1398] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:45.382730 32718 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:45.384368 32724 tablet_replica.cc:330] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:45.385908 32724 raft_consensus.cc:2229] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.387173 32724 raft_consensus.cc:2258] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.388399 32724 ts_tablet_manager.cc:1875] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.390444 32718 tablet_bootstrap.cc:654] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.396191 32718 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:45.396440 32718 ts_tablet_manager.cc:1367] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:00:45.397346 32718 raft_consensus.cc:348] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.397720 32718 raft_consensus.cc:374] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.397859 32718 raft_consensus.cc:729] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.397982 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8020272a8b14dea8a6a82802ea4e10a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:45.398276 32718 consensus_queue.cc:260] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.398607 32362 raft_consensus.cc:2454] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:45.399114 31899 leader_election.cc:302] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:00:45.399386 32208 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:45.399533 32692 raft_consensus.cc:2781] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.399657 32692 raft_consensus.cc:482] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.399747 32692 raft_consensus.cc:3037] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.400247 32726 tablet_replica.cc:330] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:45.400481 32726 raft_consensus.cc:2229] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.400751 32726 raft_consensus.cc:2258] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.401593 32726 ts_tablet_manager.cc:1875] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.402125 32660 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:45.402287 32660 ts_tablet_manager.cc:1367] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20240501 14:00:45.402719 32692 raft_consensus.cc:504] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.403360 32692 leader_election.cc:288] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:45.405969 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8020272a8b14dea8a6a82802ea4e10a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:45.406204 32362 raft_consensus.cc:3037] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.406949 32721 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:45.408721 32721 tablet_bootstrap.cc:654] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.408824 32724 ts_tablet_manager.cc:1888] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.408850 32362 raft_consensus.cc:2454] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:45.408991 32724 log.cc:1196] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/b508eaf4280143df96ef3c72d54194ee
I20240501 14:00:45.409502 32724 ts_tablet_manager.cc:1909] T b508eaf4280143df96ef3c72d54194ee P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:45.409788 31899 leader_election.cc:302] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.411262 32692 raft_consensus.cc:2781] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.411460 32692 raft_consensus.cc:686] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:45.411872 32692 consensus_queue.cc:237] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:45.413709 32718 ts_tablet_manager.cc:1398] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:00:45.416050 32660 raft_consensus.cc:348] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.416312 32660 raft_consensus.cc:374] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.416405 32660 raft_consensus.cc:729] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.416640 32660 consensus_queue.cc:260] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.421284 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet b508eaf4280143df96ef3c72d54194ee (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.423585 32660 ts_tablet_manager.cc:1398] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20240501 14:00:45.423655 29103 catalog_manager.cc:5484] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:45.425282 32460 ts_tablet_manager.cc:1888] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.430409 32460 log.cc:1196] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/b4ad2362aa0341d1b4faf9907d143458
I20240501 14:00:45.432598 32460 ts_tablet_manager.cc:1909] T b4ad2362aa0341d1b4faf9907d143458 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:45.433907 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet b4ad2362aa0341d1b4faf9907d143458 (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.405094 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8020272a8b14dea8a6a82802ea4e10a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:45.436590 29245 raft_consensus.cc:3037] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.429525 32719 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:45.437734 32096 raft_consensus.cc:1261] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.438314 32692 consensus_queue.cc:1035] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.440192 29245 raft_consensus.cc:2454] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:45.410439 32725 tablet_replica.cc:330] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.441100 32725 raft_consensus.cc:2229] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.441402 32725 raft_consensus.cc:2258] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.410985 29246 raft_consensus.cc:1261] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.442260 32725 ts_tablet_manager.cc:1875] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.442639 32719 tablet_bootstrap.cc:654] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.442652 32692 consensus_queue.cc:1035] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.455127 32721 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:45.457592 32725 ts_tablet_manager.cc:1888] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.457818 32725 log.cc:1196] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/426bc72f14fb4490b329ab52bf7ca69c
I20240501 14:00:45.458437 32725 ts_tablet_manager.cc:1909] T 426bc72f14fb4490b329ab52bf7ca69c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.459100 29103 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]). Expected version 1 got 0
I20240501 14:00:45.460201 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 426bc72f14fb4490b329ab52bf7ca69c (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.460726 32725 tablet_replica.cc:330] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.461262 32725 raft_consensus.cc:2229] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.461628 32725 raft_consensus.cc:2258] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.462806 32725 ts_tablet_manager.cc:1875] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.468921 32661 mvcc.cc:205] Tried to move back new op lower bound from 7022887097995763712 to 7022887096566263808. Current Snapshot: MvccSnapshot[applied={T|T < 7022887097995763712}]
I20240501 14:00:45.473235 32740 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.474085 32738 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.475622 29103 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]). Expected version 1 got 0
I20240501 14:00:45.476516 32738 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:45.477105 32721 ts_tablet_manager.cc:1367] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20240501 14:00:45.478025 32721 raft_consensus.cc:348] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.478240 32721 raft_consensus.cc:374] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.478320 32721 raft_consensus.cc:729] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.482431 32721 consensus_queue.cc:260] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.482851 32574 raft_consensus.cc:482] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.483031 32574 raft_consensus.cc:504] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.483783 32574 leader_election.cc:288] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:45.485540 32719 tablet_bootstrap.cc:492] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:45.486512 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b0222ebb6d4246ab8551d5579b1760" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:45.486889 32096 raft_consensus.cc:2454] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:45.487392 32296 leader_election.cc:302] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:45.483068 32721 ts_tablet_manager.cc:1398] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:45.487767 32574 raft_consensus.cc:2781] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.487881 32574 raft_consensus.cc:482] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.487989 32574 raft_consensus.cc:3037] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.495330 29100 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]). Expected version 1 got 0
I20240501 14:00:45.500703 32726 ts_tablet_manager.cc:1888] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.502876 32725 ts_tablet_manager.cc:1888] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.503129 32725 log.cc:1196] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4ad2362aa0341d1b4faf9907d143458
I20240501 14:00:45.504171 32725 ts_tablet_manager.cc:1909] T b4ad2362aa0341d1b4faf9907d143458 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.505172 32725 tablet_replica.cc:330] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.505319 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet b4ad2362aa0341d1b4faf9907d143458 (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.505506 32725 raft_consensus.cc:2229] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.505580 32726 log.cc:1196] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/b4ad2362aa0341d1b4faf9907d143458
I20240501 14:00:45.505880 32725 raft_consensus.cc:2258] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.506070 32726 ts_tablet_manager.cc:1909] T b4ad2362aa0341d1b4faf9907d143458 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:45.506685 32725 ts_tablet_manager.cc:1875] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.511124 32719 ts_tablet_manager.cc:1367] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.001s
I20240501 14:00:45.512466 32726 tablet_replica.cc:330] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:45.512625 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet b4ad2362aa0341d1b4faf9907d143458 (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.512853 32726 raft_consensus.cc:2229] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.513151 32726 raft_consensus.cc:2258] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.514066 32726 ts_tablet_manager.cc:1875] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.517737 32719 raft_consensus.cc:348] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.517989 32719 raft_consensus.cc:374] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.518092 32719 raft_consensus.cc:729] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.518332 32719 consensus_queue.cc:260] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.518942 32719 ts_tablet_manager.cc:1398] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:00:45.524020 32740 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:45.487128 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b0222ebb6d4246ab8551d5579b1760" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:45.524946 31965 raft_consensus.cc:2454] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:45.534780 32725 ts_tablet_manager.cc:1888] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.535041 32725 log.cc:1196] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/b508eaf4280143df96ef3c72d54194ee
I20240501 14:00:45.535571 32725 ts_tablet_manager.cc:1909] T b508eaf4280143df96ef3c72d54194ee P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.538637 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet b508eaf4280143df96ef3c72d54194ee (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.540668 32726 ts_tablet_manager.cc:1888] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.541585 32741 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.542758 32741 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:45.542865 32726 log.cc:1196] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/b508eaf4280143df96ef3c72d54194ee
I20240501 14:00:45.543363 32726 ts_tablet_manager.cc:1909] T b508eaf4280143df96ef3c72d54194ee P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:45.544391 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet b508eaf4280143df96ef3c72d54194ee (table rebalance_test_table_8 [id=0f9c8a900a4a4a81864dc6d0d7731896]) successfully deleted
I20240501 14:00:45.544880 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa] alter complete (version 1)
I20240501 14:00:45.547791 29103 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa]). Expected version 1 got 0
I20240501 14:00:45.556020 32574 raft_consensus.cc:504] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.556802 32574 leader_election.cc:288] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:45.557449 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b0222ebb6d4246ab8551d5579b1760" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:45.557672 31965 raft_consensus.cc:3037] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.560469 31965 raft_consensus.cc:2454] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:45.561210 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b0222ebb6d4246ab8551d5579b1760" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:45.561376 32296 leader_election.cc:302] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.561499 32096 raft_consensus.cc:3037] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.561739 32574 raft_consensus.cc:2781] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.562296 32574 raft_consensus.cc:686] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:45.562669 32574 consensus_queue.cc:237] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.564381 32096 raft_consensus.cc:2454] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:45.565740 29103 catalog_manager.cc:5484] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:45.671294 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:45.680339 32738 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.680891 32738 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:45.682943 32741 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.683408 32741 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:45.683770 32740 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.684314 32740 tablet.cc:1620] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:45.690539 29100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=65cf90100af74431b5700c6a0046dbfa] alter complete (version 2)
W20240501 14:00:45.703675 29423 tablet.cc:2369] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:45.712345 29375 consensus_queue.cc:237] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:45.714854 29244 raft_consensus.cc:1261] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.719115 32435 consensus_queue.cc:1035] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.722388 29505 raft_consensus.cc:1261] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.722959 32435 consensus_queue.cc:1035] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.725875 32762 raft_consensus.cc:2932] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:45.726676 29505 raft_consensus.cc:2932] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
W20240501 14:00:45.728565 29311 consensus_peers.cc:485] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b76fa6c27c44d7588cd0fb133113bfe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.733259 29103 catalog_manager.cc:5484] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:45.735654 29244 raft_consensus.cc:2932] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:45.766872 32695 raft_consensus.cc:482] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.767087 32695 raft_consensus.cc:504] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.768289 32695 leader_election.cc:288] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:45.768940 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdebc6e2fdd4b2daa399cd1301fcde8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:45.769246 32227 raft_consensus.cc:2454] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:45.769748 31899 leader_election.cc:302] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.770032 32695 raft_consensus.cc:2781] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.770156 32695 raft_consensus.cc:482] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.770251 32695 raft_consensus.cc:3037] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.772207 32695 raft_consensus.cc:504] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.773613 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdebc6e2fdd4b2daa399cd1301fcde8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:45.773860 32227 raft_consensus.cc:3037] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.776320 32227 raft_consensus.cc:2454] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:45.776768 31899 leader_election.cc:302] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:45.777070 32692 raft_consensus.cc:2781] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.777031 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdebc6e2fdd4b2daa399cd1301fcde8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:45.777035 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdebc6e2fdd4b2daa399cd1301fcde8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:45.777227 32692 raft_consensus.cc:686] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:45.777288 29375 raft_consensus.cc:2454] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:45.777657 32695 leader_election.cc:288] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:45.777521 32692 consensus_queue.cc:237] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.788964 29100 catalog_manager.cc:5484] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:45.792066 29244 consensus_queue.cc:237] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:45.794389 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:45.794828 29637 raft_consensus.cc:1261] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.795897 32729 consensus_queue.cc:1035] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.804236 32738 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.805490 32740 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.809201 29505 raft_consensus.cc:1261] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.810392 32729 consensus_queue.cc:1035] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.813529 32741 tablet.cc:1722] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.816192 29100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=65cf90100af74431b5700c6a0046dbfa] alter complete (version 3)
I20240501 14:00:45.819938 32602 raft_consensus.cc:2932] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:45.824478 29100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:00:45.824731 29100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:00:45.825968 29637 raft_consensus.cc:2932] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:45.830094 29504 raft_consensus.cc:2932] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:45.832836 29100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 01ebbac3d0aa4c23b42157d5c0d2acc9
I20240501 14:00:45.833596 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:45.833905 32726 tablet_replica.cc:330] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:45.834120 32726 raft_consensus.cc:2229] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.834385 32726 raft_consensus.cc:2258] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.835122 32726 ts_tablet_manager.cc:1875] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.836084 29100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 954bd347d6944d9cbc7587cb2435487e
I20240501 14:00:45.837683 29100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 8227f23eac39422eb5fa519d202a0d57
I20240501 14:00:45.838102 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 954bd347d6944d9cbc7587cb2435487e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:45.838394 32724 tablet_replica.cc:330] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:45.838438 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:45.838693 32460 tablet_replica.cc:330] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:45.838887 32460 raft_consensus.cc:2229] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.839143 32460 raft_consensus.cc:2258] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.839304 32724 raft_consensus.cc:2229] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.839291 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8227f23eac39422eb5fa519d202a0d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:45.839612 32724 raft_consensus.cc:2258] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.839763 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 8227f23eac39422eb5fa519d202a0d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:45.839910 32460 ts_tablet_manager.cc:1875] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.840689 32724 ts_tablet_manager.cc:1875] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.841092 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 954bd347d6944d9cbc7587cb2435487e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:45.846282 32460 ts_tablet_manager.cc:1888] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.849129 32460 log.cc:1196] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/01ebbac3d0aa4c23b42157d5c0d2acc9
I20240501 14:00:45.849498 32726 ts_tablet_manager.cc:1888] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.849612 32460 ts_tablet_manager.cc:1909] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:45.849773 32726 log.cc:1196] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/01ebbac3d0aa4c23b42157d5c0d2acc9
I20240501 14:00:45.850224 32726 ts_tablet_manager.cc:1909] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:45.850829 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.851277 32726 tablet_replica.cc:330] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:45.851372 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.851593 32460 tablet_replica.cc:330] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:45.851622 32726 raft_consensus.cc:2229] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.851891 32460 raft_consensus.cc:2229] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.852056 32726 raft_consensus.cc:2258] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.852838 32726 ts_tablet_manager.cc:1875] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.853363 32460 raft_consensus.cc:2258] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.854075 32460 ts_tablet_manager.cc:1875] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.855208 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 954bd347d6944d9cbc7587cb2435487e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.855491 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.855732 32725 tablet_replica.cc:330] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.855967 32725 raft_consensus.cc:2229] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.856335 32725 raft_consensus.cc:2258] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.857365 32725 ts_tablet_manager.cc:1875] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.860067 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 954bd347d6944d9cbc7587cb2435487e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.860654 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 8227f23eac39422eb5fa519d202a0d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.864675 32724 ts_tablet_manager.cc:1888] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.866816 32460 ts_tablet_manager.cc:1888] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.866830 32724 log.cc:1196] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/954bd347d6944d9cbc7587cb2435487e
I20240501 14:00:45.866976 32460 log.cc:1196] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/8227f23eac39422eb5fa519d202a0d57
I20240501 14:00:45.867394 32460 ts_tablet_manager.cc:1909] T 8227f23eac39422eb5fa519d202a0d57 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:45.867682 32724 ts_tablet_manager.cc:1909] T 954bd347d6944d9cbc7587cb2435487e P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:45.869280 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 954bd347d6944d9cbc7587cb2435487e (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.872291 32725 ts_tablet_manager.cc:1888] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.872435 32725 log.cc:1196] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/954bd347d6944d9cbc7587cb2435487e
I20240501 14:00:45.872835 32725 ts_tablet_manager.cc:1909] T 954bd347d6944d9cbc7587cb2435487e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.874831 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 954bd347d6944d9cbc7587cb2435487e (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.876152 32725 tablet_replica.cc:330] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.876811 32725 raft_consensus.cc:2229] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.877224 32725 raft_consensus.cc:2258] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.878060 32725 ts_tablet_manager.cc:1875] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.878566 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 8227f23eac39422eb5fa519d202a0d57 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.887248 32725 ts_tablet_manager.cc:1888] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.887390 32725 log.cc:1196] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/01ebbac3d0aa4c23b42157d5c0d2acc9
I20240501 14:00:45.887882 32725 ts_tablet_manager.cc:1909] T 01ebbac3d0aa4c23b42157d5c0d2acc9 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.889120 32726 ts_tablet_manager.cc:1888] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.889110 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 01ebbac3d0aa4c23b42157d5c0d2acc9 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.889266 32725 tablet_replica.cc:330] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
W20240501 14:00:45.889678 29089 catalog_manager.cc:4797] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): delete failed for tablet 954bd347d6944d9cbc7587cb2435487e because the tablet was not found. No further retry: Not found: Tablet not found: 954bd347d6944d9cbc7587cb2435487e
I20240501 14:00:45.889806 32726 log.cc:1196] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/8227f23eac39422eb5fa519d202a0d57
I20240501 14:00:45.889762 32725 raft_consensus.cc:2229] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.890079 32725 raft_consensus.cc:2258] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.890823 32726 ts_tablet_manager.cc:1909] T 8227f23eac39422eb5fa519d202a0d57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:45.890849 32725 ts_tablet_manager.cc:1875] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.891834 32726 tablet_replica.cc:330] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:45.892040 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 8227f23eac39422eb5fa519d202a0d57 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.892095 32726 raft_consensus.cc:2229] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.892419 32726 raft_consensus.cc:2258] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.893417 32726 ts_tablet_manager.cc:1875] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.897954 32725 ts_tablet_manager.cc:1888] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:45.898208 32725 log.cc:1196] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/8227f23eac39422eb5fa519d202a0d57
I20240501 14:00:45.898697 32725 ts_tablet_manager.cc:1909] T 8227f23eac39422eb5fa519d202a0d57 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.899859 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 8227f23eac39422eb5fa519d202a0d57 (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.901966 32726 ts_tablet_manager.cc:1888] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.902086 32726 log.cc:1196] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/954bd347d6944d9cbc7587cb2435487e
I20240501 14:00:45.902943 32726 ts_tablet_manager.cc:1909] T 954bd347d6944d9cbc7587cb2435487e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:45.904731 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 954bd347d6944d9cbc7587cb2435487e (table rebalance_test_table_10 [id=71f13072514a41e9be48406a0812b4ab]) successfully deleted
I20240501 14:00:45.917440 32695 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.922458 32479 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.923725 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:45.923985 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:45.923921 29244 consensus_queue.cc:237] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:45.930546 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet d3b69db836da4f3193717e0b8c9f5ab3
I20240501 14:00:45.931754 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:45.931807 32075 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:45.934782 339 tablet_replica.cc:330] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:45.934998 339 raft_consensus.cc:2229] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.935441 339 raft_consensus.cc:2258] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.936060 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b69db836da4f3193717e0b8c9f5ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:45.936429 339 ts_tablet_manager.cc:1875] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.936455 32725 tablet_replica.cc:330] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:45.936698 32725 raft_consensus.cc:2229] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.936933 32725 raft_consensus.cc:2258] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.938488 32725 ts_tablet_manager.cc:1875] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.943460 339 ts_tablet_manager.cc:1888] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.943593 339 log.cc:1196] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/d3b69db836da4f3193717e0b8c9f5ab3
I20240501 14:00:45.944043 339 ts_tablet_manager.cc:1909] T d3b69db836da4f3193717e0b8c9f5ab3 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:45.944768 32725 ts_tablet_manager.cc:1888] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.944892 32725 log.cc:1196] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3b69db836da4f3193717e0b8c9f5ab3
I20240501 14:00:45.945243 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table_new_name [id=65cf90100af74431b5700c6a0046dbfa]) successfully deleted
I20240501 14:00:45.945513 32725 ts_tablet_manager.cc:1909] T d3b69db836da4f3193717e0b8c9f5ab3 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:45.946338 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table_new_name [id=65cf90100af74431b5700c6a0046dbfa]) successfully deleted
I20240501 14:00:45.948801 340 tablet_replica.cc:330] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:45.949318 340 raft_consensus.cc:2229] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.949620 340 raft_consensus.cc:2258] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.951076 340 ts_tablet_manager.cc:1875] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.960094 340 ts_tablet_manager.cc:1888] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.960224 29505 raft_consensus.cc:1261] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.960350 340 log.cc:1196] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/d3b69db836da4f3193717e0b8c9f5ab3
I20240501 14:00:45.961032 340 ts_tablet_manager.cc:1909] T d3b69db836da4f3193717e0b8c9f5ab3 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:45.961161 29637 raft_consensus.cc:1261] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.961853 32602 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.962389 32729 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.964397 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet d3b69db836da4f3193717e0b8c9f5ab3 (table rebalancer_dynamic_table_new_name [id=65cf90100af74431b5700c6a0046dbfa]) successfully deleted
I20240501 14:00:45.969202 32602 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:45.969961 29637 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
W20240501 14:00:45.970424 29181 consensus_peers.cc:485] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60527f81f34b45c199bf52bf0df53ee2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.973974 29505 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:45.978809 29100 catalog_manager.cc:5484] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:46.014056 29505 consensus_queue.cc:237] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.016697 29376 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:46.017324 29637 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:46.017786 29442 consensus_peers.cc:485] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:46.018201 32279 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:46.018636 32764 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:46.033543 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:46.042765 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 8763640f71a3446e9a82f9387ec789f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]), partition=
I20240501 14:00:46.043220 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8763640f71a3446e9a82f9387ec789f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.043363 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 8763640f71a3446e9a82f9387ec789f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]), partition=
I20240501 14:00:46.043720 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8763640f71a3446e9a82f9387ec789f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.044487 32279 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:46.047525 29376 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:46.048923 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 8763640f71a3446e9a82f9387ec789f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]), partition=
I20240501 14:00:46.049311 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8763640f71a3446e9a82f9387ec789f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.051898 29635 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:46.053752 29103 catalog_manager.cc:5484] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:46.055212 32574 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.073544 368 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:46.081902 368 tablet_bootstrap.cc:654] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.089597 367 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:46.095343 368 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:46.096444 368 ts_tablet_manager.cc:1367] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240501 14:00:46.097877 368 raft_consensus.cc:348] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.099792 368 raft_consensus.cc:374] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.100150 368 raft_consensus.cc:729] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.100690 368 consensus_queue.cc:260] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.103235 368 ts_tablet_manager.cc:1398] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:46.126715 32574 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.128116 367 tablet_bootstrap.cc:654] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.128244 373 ts_tablet_manager.cc:902] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:46.133730 364 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:46.138803 364 tablet_bootstrap.cc:654] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.138923 373 tablet_copy_client.cc:308] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:00:46.140352 367 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:46.140583 367 ts_tablet_manager.cc:1367] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20240501 14:00:46.141551 367 raft_consensus.cc:348] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.141856 367 raft_consensus.cc:374] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.142014 367 raft_consensus.cc:729] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.142511 367 consensus_queue.cc:260] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.143260 367 ts_tablet_manager.cc:1398] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:46.154839 364 tablet_bootstrap.cc:492] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:46.155033 364 ts_tablet_manager.cc:1367] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240501 14:00:46.155766 364 raft_consensus.cc:348] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.155956 364 raft_consensus.cc:374] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.156059 364 raft_consensus.cc:729] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.156315 364 consensus_queue.cc:260] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.156888 364 ts_tablet_manager.cc:1398] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.164731 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 9b76fa6c27c44d7588cd0fb133113bfe from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:49865)
I20240501 14:00:46.165016 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 9b76fa6c27c44d7588cd0fb133113bfe from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:49865: session id = 6edd6a3430d44c38a19c2804e555059f-9b76fa6c27c44d7588cd0fb133113bfe
I20240501 14:00:46.169303 29397 tablet_copy_source_session.cc:205] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.172096 373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b76fa6c27c44d7588cd0fb133113bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.177188 32433 raft_consensus.cc:482] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.177392 32433 raft_consensus.cc:504] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.178238 32433 leader_election.cc:288] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:46.178939 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8763640f71a3446e9a82f9387ec789f8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:46.179200 31965 raft_consensus.cc:2454] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:46.179816 32163 leader_election.cc:302] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:46.180209 32433 raft_consensus.cc:2781] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.180315 32433 raft_consensus.cc:482] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.180425 32433 raft_consensus.cc:3037] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.185262 32433 raft_consensus.cc:504] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.186499 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8763640f71a3446e9a82f9387ec789f8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:46.186739 31965 raft_consensus.cc:3037] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.187806 373 tablet_copy_client.cc:783] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.188917 373 tablet_copy_client.cc:647] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.189421 31965 raft_consensus.cc:2454] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:46.190071 32163 leader_election.cc:302] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:46.191404 32433 leader_election.cc:288] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:46.191792 32433 raft_consensus.cc:2781] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.192174 32433 raft_consensus.cc:686] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:46.192672 32433 consensus_queue.cc:237] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.192761 373 tablet_copy_client.cc:523] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.197445 29103 catalog_manager.cc:5484] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:46.198498 373 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:46.207105 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8763640f71a3446e9a82f9387ec789f8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:46.207353 32362 raft_consensus.cc:2454] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:46.207759 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8763640f71a3446e9a82f9387ec789f8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:46.207983 32361 raft_consensus.cc:3037] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.213591 32361 raft_consensus.cc:2454] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:46.256907 373 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.258121 373 tablet_bootstrap.cc:492] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:00:46.258522 373 ts_tablet_manager.cc:1367] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.060s user 0.049s sys 0.010s
I20240501 14:00:46.259636 373 raft_consensus.cc:348] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:46.260017 373 raft_consensus.cc:729] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.260476 373 consensus_queue.cc:260] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:46.261389 373 ts_tablet_manager.cc:1398] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240501 14:00:46.262560 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-9b76fa6c27c44d7588cd0fb133113bfe received from {username='slave'} at 127.22.33.72:49865
I20240501 14:00:46.262771 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-9b76fa6c27c44d7588cd0fb133113bfe on tablet 9b76fa6c27c44d7588cd0fb133113bfe with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:46.303676 29376 raft_consensus.cc:3037] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.306651 32479 consensus_queue.cc:1035] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.326683 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:46.328545 29104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:46.333299 32695 consensus_queue.cc:1035] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.354077 31965 raft_consensus.cc:1261] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.356081 387 consensus_queue.cc:1035] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.359594 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:00:46.359951 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:00:46.367753 32075 tablet_service.cc:1467] Processing CreateTablet for tablet ad7af304700545b2a28249f9f046cab6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.367982 32207 tablet_service.cc:1467] Processing CreateTablet for tablet ad7af304700545b2a28249f9f046cab6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.368299 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7af304700545b2a28249f9f046cab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.374058 29105 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]). Expected version 1 got 0
I20240501 14:00:46.374812 32342 tablet_service.cc:1467] Processing CreateTablet for tablet ad7af304700545b2a28249f9f046cab6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.375275 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7af304700545b2a28249f9f046cab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.379235 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 4 replicas of tablet 9b76fa6c27c44d7588cd0fb133113bfe
I20240501 14:00:46.380136 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet c451da0cce1447d2aa3209a8837b82a4
I20240501 14:00:46.380440 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:46.380932 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:46.381112 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.381384 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b76fa6c27c44d7588cd0fb133113bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:46.381464 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet c451da0cce1447d2aa3209a8837b82a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:46.381767 32726 tablet_replica.cc:330] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:46.381815 32725 tablet_replica.cc:330] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.382022 32726 raft_consensus.cc:2229] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.382022 32725 raft_consensus.cc:2229] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.382266 32726 raft_consensus.cc:2258] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.382336 32725 raft_consensus.cc:2258] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.383106 32726 ts_tablet_manager.cc:1875] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.383401 32725 ts_tablet_manager.cc:1875] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.387042 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 62c84a393bd047d985927417f82bec7d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.387535 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c84a393bd047d985927417f82bec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.389097 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 09c1d1f40c8f4eaca06a3dc9c0304dce
I20240501 14:00:46.389894 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet c451da0cce1447d2aa3209a8837b82a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:46.389997 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:46.390202 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet c451da0cce1447d2aa3209a8837b82a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.391770 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:46.394649 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c1d1f40c8f4eaca06a3dc9c0304dce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.396319 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 62c84a393bd047d985927417f82bec7d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.396829 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c84a393bd047d985927417f82bec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.398036 32725 ts_tablet_manager.cc:1888] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.398212 32725 log.cc:1196] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b76fa6c27c44d7588cd0fb133113bfe
I20240501 14:00:46.398661 32725 ts_tablet_manager.cc:1909] T 9b76fa6c27c44d7588cd0fb133113bfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.399507 32726 ts_tablet_manager.cc:1888] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.400236 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 9b76fa6c27c44d7588cd0fb133113bfe (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.400542 32726 log.cc:1196] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/c451da0cce1447d2aa3209a8837b82a4
I20240501 14:00:46.400794 32725 tablet_replica.cc:330] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.401722 32725 raft_consensus.cc:2229] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.402096 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 768543d354744b44acdebb9a95a1579b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.401530 32726 ts_tablet_manager.cc:1909] T c451da0cce1447d2aa3209a8837b82a4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:46.402544 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768543d354744b44acdebb9a95a1579b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.404883 32726 tablet_replica.cc:330] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:46.405917 32726 raft_consensus.cc:2229] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.406247 32726 raft_consensus.cc:2258] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.409080 364 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:46.411253 364 tablet_bootstrap.cc:654] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.417227 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 768543d354744b44acdebb9a95a1579b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.417685 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768543d354744b44acdebb9a95a1579b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.419521 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet c451da0cce1447d2aa3209a8837b82a4 (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.424580 32726 ts_tablet_manager.cc:1875] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.427268 367 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:46.429613 367 tablet_bootstrap.cc:654] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.437386 32725 raft_consensus.cc:2258] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.440431 367 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:46.440716 367 ts_tablet_manager.cc:1367] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240501 14:00:46.441911 367 raft_consensus.cc:348] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.442148 32725 ts_tablet_manager.cc:1875] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.442396 367 raft_consensus.cc:374] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.442562 367 raft_consensus.cc:729] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.442993 367 consensus_queue.cc:260] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.444764 367 ts_tablet_manager.cc:1398] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:46.446110 364 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:46.446369 364 ts_tablet_manager.cc:1367] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.006s
I20240501 14:00:46.447360 364 raft_consensus.cc:348] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.449115 364 raft_consensus.cc:374] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.449271 364 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.449636 364 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.450896 364 ts_tablet_manager.cc:1398] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:46.451157 426 tablet_replica.cc:330] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:46.451550 426 raft_consensus.cc:2229] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.452399 32361 raft_consensus.cc:1261] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.453272 427 tablet_replica.cc:330] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:46.453531 427 raft_consensus.cc:2229] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.455439 426 raft_consensus.cc:2258] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.455482 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7af304700545b2a28249f9f046cab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.455956 364 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:46.456024 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 62c84a393bd047d985927417f82bec7d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.457279 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c84a393bd047d985927417f82bec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.457916 427 raft_consensus.cc:2258] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.461639 364 tablet_bootstrap.cc:654] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.462524 426 ts_tablet_manager.cc:1875] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.462749 427 ts_tablet_manager.cc:1875] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.475458 443 ts_tablet_manager.cc:902] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:46.480715 32692 raft_consensus.cc:482] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.480926 32692 raft_consensus.cc:504] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.483773 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:00:46.484516 31900 leader_election.cc:341] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.484952 32692 leader_election.cc:288] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:46.491873 428 tablet_replica.cc:330] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:46.492154 428 raft_consensus.cc:2229] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.492439 428 raft_consensus.cc:2258] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.492712 430 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:46.494709 430 tablet_bootstrap.cc:654] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.495332 29487 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:46.497627 428 ts_tablet_manager.cc:1875] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.500866 32725 ts_tablet_manager.cc:1888] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.501180 32725 log.cc:1196] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/c451da0cce1447d2aa3209a8837b82a4
I20240501 14:00:46.501849 32725 ts_tablet_manager.cc:1909] T c451da0cce1447d2aa3209a8837b82a4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.503165 32725 tablet_replica.cc:330] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.503489 32725 raft_consensus.cc:2229] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.507511 427 ts_tablet_manager.cc:1888] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.513496 364 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:46.513746 364 ts_tablet_manager.cc:1367] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240501 14:00:46.515020 364 raft_consensus.cc:348] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.515328 364 raft_consensus.cc:374] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.515496 364 raft_consensus.cc:729] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.515805 364 consensus_queue.cc:260] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.516453 364 ts_tablet_manager.cc:1398] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:46.520489 32725 raft_consensus.cc:2258] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.521571 32725 ts_tablet_manager.cc:1875] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.525712 32726 ts_tablet_manager.cc:1888] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.527575 32726 log.cc:1196] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/09c1d1f40c8f4eaca06a3dc9c0304dce
I20240501 14:00:46.528458 32726 ts_tablet_manager.cc:1909] T 09c1d1f40c8f4eaca06a3dc9c0304dce P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:46.530680 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.531193 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet c451da0cce1447d2aa3209a8837b82a4 (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.536442 428 ts_tablet_manager.cc:1888] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.538518 427 log.cc:1196] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b76fa6c27c44d7588cd0fb133113bfe
I20240501 14:00:46.539196 427 ts_tablet_manager.cc:1909] T 9b76fa6c27c44d7588cd0fb133113bfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:46.540503 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 9b76fa6c27c44d7588cd0fb133113bfe (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.540951 427 tablet_replica.cc:330] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:46.541235 426 ts_tablet_manager.cc:1888] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.541277 427 raft_consensus.cc:2229] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.541576 427 raft_consensus.cc:2258] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.543402 427 ts_tablet_manager.cc:1875] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.545285 29226 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:46.564428 426 log.cc:1196] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b76fa6c27c44d7588cd0fb133113bfe
I20240501 14:00:46.565448 426 ts_tablet_manager.cc:1909] T 9b76fa6c27c44d7588cd0fb133113bfe P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:46.566725 442 consensus_queue.cc:1035] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.567798 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 768543d354744b44acdebb9a95a1579b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.568323 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768543d354744b44acdebb9a95a1579b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.571609 428 log.cc:1196] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/9b76fa6c27c44d7588cd0fb133113bfe
W20240501 14:00:46.572047 31899 leader_election.cc:341] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Not found: Tablet not found: 768543d354744b44acdebb9a95a1579b
I20240501 14:00:46.572125 428 ts_tablet_manager.cc:1909] T 9b76fa6c27c44d7588cd0fb133113bfe P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:46.572190 31899 leader_election.cc:302] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:46.573231 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 9b76fa6c27c44d7588cd0fb133113bfe (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.573530 426 tablet_replica.cc:330] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:46.573786 426 raft_consensus.cc:2229] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.574041 426 raft_consensus.cc:2258] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.574793 426 ts_tablet_manager.cc:1875] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.574870 32692 raft_consensus.cc:2726] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:46.575282 451 ts_tablet_manager.cc:902] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:46.579524 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 9b76fa6c27c44d7588cd0fb133113bfe (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.580153 368 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:46.580314 32725 ts_tablet_manager.cc:1888] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.580467 32725 log.cc:1196] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/09c1d1f40c8f4eaca06a3dc9c0304dce
I20240501 14:00:46.580940 32725 ts_tablet_manager.cc:1909] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.581952 368 tablet_bootstrap.cc:654] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.586025 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.588919 443 tablet_copy_client.cc:308] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:00:46.589644 433 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.590368 433 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:46.598910 427 ts_tablet_manager.cc:1888] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.599797 32341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:46.604097 427 log.cc:1196] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/09c1d1f40c8f4eaca06a3dc9c0304dce
I20240501 14:00:46.605079 427 ts_tablet_manager.cc:1909] T 09c1d1f40c8f4eaca06a3dc9c0304dce P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:46.610970 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 09c1d1f40c8f4eaca06a3dc9c0304dce (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.611927 426 ts_tablet_manager.cc:1888] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.612097 426 log.cc:1196] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/c451da0cce1447d2aa3209a8837b82a4
I20240501 14:00:46.612522 426 ts_tablet_manager.cc:1909] T c451da0cce1447d2aa3209a8837b82a4 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:46.613579 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet c451da0cce1447d2aa3209a8837b82a4 (table rebalance_test_table_12 [id=fb10ee2f136e4bb0a8471d277249bf0a]) successfully deleted
I20240501 14:00:46.620604 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 60527f81f34b45c199bf52bf0df53ee2 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:44499)
I20240501 14:00:46.620891 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 60527f81f34b45c199bf52bf0df53ee2 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:44499: session id = 9903ed35bca144bcb74e9bf5102117e7-60527f81f34b45c199bf52bf0df53ee2
I20240501 14:00:46.621681 368 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:46.621868 368 ts_tablet_manager.cc:1367] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.009s
I20240501 14:00:46.622853 368 raft_consensus.cc:348] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.623157 368 raft_consensus.cc:374] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.623239 368 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.623538 368 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.624152 368 ts_tablet_manager.cc:1398] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.624409 368 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:46.640050 29266 tablet_copy_source_session.cc:205] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.646559 443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60527f81f34b45c199bf52bf0df53ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.656807 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3] alter complete (version 1)
I20240501 14:00:46.656894 368 tablet_bootstrap.cc:654] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.659773 451 tablet_copy_client.cc:308] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:00:46.664487 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 49b72e2ac9df4426b071a7e5004e12e1 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36525)
I20240501 14:00:46.664744 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 49b72e2ac9df4426b071a7e5004e12e1 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36525: session id = 0a697f3a1ee544daa7073814ab845d58-49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:00:46.668920 29527 tablet_copy_source_session.cc:205] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.674875 466 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.676107 466 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:46.679884 443 tablet_copy_client.cc:783] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.680334 443 tablet_copy_client.cc:647] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.684175 29103 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]). Expected version 1 got 0
I20240501 14:00:46.690918 451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b72e2ac9df4426b071a7e5004e12e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.693356 31945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:46.699877 430 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:46.700052 430 ts_tablet_manager.cc:1367] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.250s user 0.017s sys 0.010s
I20240501 14:00:46.700876 430 raft_consensus.cc:348] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.701115 430 raft_consensus.cc:374] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.701222 430 raft_consensus.cc:729] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.701493 430 consensus_queue.cc:260] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.702061 430 ts_tablet_manager.cc:1398] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:46.710345 430 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:46.712415 430 tablet_bootstrap.cc:654] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.723902 443 tablet_copy_client.cc:523] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.724315 451 tablet_copy_client.cc:783] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.724628 451 tablet_copy_client.cc:647] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.728359 443 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:46.729156 371 raft_consensus.cc:482] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.729352 371 raft_consensus.cc:504] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.730106 371 leader_election.cc:288] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:46.731006 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:46.731698 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
W20240501 14:00:46.731803 32296 leader_election.cc:341] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:46.732288 32295 leader_election.cc:341] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.732419 32295 leader_election.cc:302] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:46.733074 371 raft_consensus.cc:2726] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:46.737277 29357 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:46.750522 430 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:46.750738 430 ts_tablet_manager.cc:1367] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.001s
I20240501 14:00:46.751789 430 raft_consensus.cc:348] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.755028 469 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.759541 29103 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]). Expected version 1 got 0
I20240501 14:00:46.760025 32641 raft_consensus.cc:482] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.760267 32641 raft_consensus.cc:504] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.761201 32641 leader_election.cc:288] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:46.761310 469 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:46.761543 430 raft_consensus.cc:374] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.761658 430 raft_consensus.cc:729] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.762125 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:46.762369 31964 raft_consensus.cc:2454] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:46.763324 32032 leader_election.cc:302] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:46.761924 430 consensus_queue.cc:260] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.763758 32641 raft_consensus.cc:2781] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.763846 32641 raft_consensus.cc:482] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.763962 32641 raft_consensus.cc:3037] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.767239 32641 raft_consensus.cc:504] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.771101 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:46.771373 31964 raft_consensus.cc:3037] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.774960 31964 raft_consensus.cc:2454] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:46.775595 32032 leader_election.cc:302] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:46.777490 32641 leader_election.cc:288] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:46.779047 32641 raft_consensus.cc:2781] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.779458 32641 raft_consensus.cc:686] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:46.779544 29615 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:46.779935 32641 consensus_queue.cc:237] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.784649 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:46.791260 29103 catalog_manager.cc:5484] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:46.796743 368 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:46.796936 368 ts_tablet_manager.cc:1367] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.000s
I20240501 14:00:46.797822 368 raft_consensus.cc:348] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.798031 368 raft_consensus.cc:374] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.798121 368 raft_consensus.cc:729] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.798338 368 consensus_queue.cc:260] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.798880 368 ts_tablet_manager.cc:1398] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.799172 368 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:46.800892 368 tablet_bootstrap.cc:654] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.807360 32208 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:46.809072 433 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.809581 433 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:46.811097 32077 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:46.811033 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:46.811573 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:46.811789 451 tablet_copy_client.cc:523] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.815953 451 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:46.822795 466 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.823292 466 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
W20240501 14:00:46.832654 32031 leader_election.cc:341] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:46.833343 32031 leader_election.cc:341] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.833609 29101 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3]). Expected version 2 got 1
I20240501 14:00:46.837392 430 ts_tablet_manager.cc:1398] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.087s user 0.006s sys 0.018s
I20240501 14:00:46.838137 430 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:46.839780 430 tablet_bootstrap.cc:654] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.838832 469 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.846311 469 tablet.cc:1620] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:46.854283 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=74688fc9834142a1ae7aa6a2e5bcaac3] alter complete (version 2)
I20240501 14:00:46.867776 368 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:46.867985 368 ts_tablet_manager.cc:1367] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.013s
I20240501 14:00:46.868937 368 raft_consensus.cc:348] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.869218 368 raft_consensus.cc:374] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.869319 368 raft_consensus.cc:729] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.869619 368 consensus_queue.cc:260] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.871214 368 ts_tablet_manager.cc:1398] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:46.894558 430 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:46.894757 430 ts_tablet_manager.cc:1367] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20240501 14:00:46.895614 430 raft_consensus.cc:348] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.895675 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:00:46.895848 430 raft_consensus.cc:374] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.895941 430 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.895903 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:00:46.896191 430 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.896771 430 ts_tablet_manager.cc:1398] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:46.901235 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet e670a438f0a94bc0814536c5a8ecbf9e
I20240501 14:00:46.902786 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.903012 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:46.903062 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet b679ed4f63414ae1aefe8cd769b4b05c
I20240501 14:00:46.903112 32725 tablet_replica.cc:330] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.903316 426 tablet_replica.cc:330] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:46.903333 32725 raft_consensus.cc:2229] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.903343 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet e670a438f0a94bc0814536c5a8ecbf9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:46.903525 426 raft_consensus.cc:2229] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.903595 32726 tablet_replica.cc:330] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:46.903614 32725 raft_consensus.cc:2258] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.903800 32726 raft_consensus.cc:2229] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.903877 426 raft_consensus.cc:2258] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.904033 32726 raft_consensus.cc:2258] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.904371 32725 ts_tablet_manager.cc:1875] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.904590 426 ts_tablet_manager.cc:1875] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.904691 32726 ts_tablet_manager.cc:1875] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.910157 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet db7077d63a534acc94c56c2b6e7b1856
I20240501 14:00:46.910327 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:46.910632 427 tablet_replica.cc:330] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:46.910831 427 raft_consensus.cc:2229] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.911046 427 raft_consensus.cc:2258] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.911190 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet db7077d63a534acc94c56c2b6e7b1856 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:46.911777 427 ts_tablet_manager.cc:1875] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.912961 426 ts_tablet_manager.cc:1888] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.913112 426 log.cc:1196] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/e670a438f0a94bc0814536c5a8ecbf9e
I20240501 14:00:46.913571 426 ts_tablet_manager.cc:1909] T e670a438f0a94bc0814536c5a8ecbf9e P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:46.914678 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet e670a438f0a94bc0814536c5a8ecbf9e (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.914891 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.915372 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet db7077d63a534acc94c56c2b6e7b1856 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:46.917212 426 tablet_replica.cc:330] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:46.917440 426 raft_consensus.cc:2229] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.917670 426 raft_consensus.cc:2258] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.918385 426 ts_tablet_manager.cc:1875] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.922806 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet b679ed4f63414ae1aefe8cd769b4b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:46.923368 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet db7077d63a534acc94c56c2b6e7b1856 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:46.924820 32726 ts_tablet_manager.cc:1888] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.924837 427 ts_tablet_manager.cc:1888] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.924990 32726 log.cc:1196] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/e670a438f0a94bc0814536c5a8ecbf9e
I20240501 14:00:46.925426 32725 ts_tablet_manager.cc:1888] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.925469 32726 ts_tablet_manager.cc:1909] T e670a438f0a94bc0814536c5a8ecbf9e P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:46.925566 32725 log.cc:1196] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/e670a438f0a94bc0814536c5a8ecbf9e
I20240501 14:00:46.926024 32725 ts_tablet_manager.cc:1909] T e670a438f0a94bc0814536c5a8ecbf9e P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.926550 32726 tablet_replica.cc:330] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:46.926646 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet e670a438f0a94bc0814536c5a8ecbf9e (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.926751 32726 raft_consensus.cc:2229] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.926951 32725 tablet_replica.cc:330] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.926977 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet e670a438f0a94bc0814536c5a8ecbf9e (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.927088 32726 raft_consensus.cc:2258] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.927191 32725 raft_consensus.cc:2229] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.927395 32725 raft_consensus.cc:2258] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.927798 32726 ts_tablet_manager.cc:1875] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.928047 32725 ts_tablet_manager.cc:1875] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.932158 427 log.cc:1196] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/b679ed4f63414ae1aefe8cd769b4b05c
I20240501 14:00:46.932677 427 ts_tablet_manager.cc:1909] T b679ed4f63414ae1aefe8cd769b4b05c P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:46.933987 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet b679ed4f63414ae1aefe8cd769b4b05c (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.938441 32726 ts_tablet_manager.cc:1888] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.938586 32726 log.cc:1196] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/b679ed4f63414ae1aefe8cd769b4b05c
I20240501 14:00:46.939043 32726 ts_tablet_manager.cc:1909] T b679ed4f63414ae1aefe8cd769b4b05c P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:46.939996 32726 tablet_replica.cc:330] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:46.940217 32726 raft_consensus.cc:2229] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.940279 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet b679ed4f63414ae1aefe8cd769b4b05c (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.940506 32726 raft_consensus.cc:2258] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.941314 32726 ts_tablet_manager.cc:1875] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.952345 443 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.952850 443 tablet_bootstrap.cc:492] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:00:46.955446 32725 ts_tablet_manager.cc:1888] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.957217 426 ts_tablet_manager.cc:1888] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.957420 426 log.cc:1196] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/db7077d63a534acc94c56c2b6e7b1856
I20240501 14:00:46.957891 426 ts_tablet_manager.cc:1909] T db7077d63a534acc94c56c2b6e7b1856 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:46.959306 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet db7077d63a534acc94c56c2b6e7b1856 (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.960310 471 raft_consensus.cc:482] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.960506 471 raft_consensus.cc:504] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.960613 32725 log.cc:1196] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/b679ed4f63414ae1aefe8cd769b4b05c
I20240501 14:00:46.961267 32725 ts_tablet_manager.cc:1909] T b679ed4f63414ae1aefe8cd769b4b05c P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.961400 471 leader_election.cc:288] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:46.962210 32725 tablet_replica.cc:330] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:46.962379 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet b679ed4f63414ae1aefe8cd769b4b05c (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.962662 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:46.962668 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:46.962929 32228 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:46.962929 31964 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:46.963346 32031 leader_election.cc:302] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:46.964021 471 raft_consensus.cc:2781] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.964150 471 raft_consensus.cc:482] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.964254 471 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.967388 471 raft_consensus.cc:504] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.967926 32725 raft_consensus.cc:2229] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.968243 32725 raft_consensus.cc:2258] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.968391 434 raft_consensus.cc:482] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.968549 434 raft_consensus.cc:504] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.969081 32725 ts_tablet_manager.cc:1875] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.969215 434 leader_election.cc:288] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:46.970326 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:46.970562 32361 raft_consensus.cc:2454] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:46.970921 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:46.971139 32096 raft_consensus.cc:2454] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:46.971499 32162 leader_election.cc:302] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:46.972395 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:46.973177 434 raft_consensus.cc:2781] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.973296 434 raft_consensus.cc:482] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.973380 434 raft_consensus.cc:3037] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.975522 32725 ts_tablet_manager.cc:1888] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.975641 32725 log.cc:1196] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/db7077d63a534acc94c56c2b6e7b1856
I20240501 14:00:46.976063 32725 ts_tablet_manager.cc:1909] T db7077d63a534acc94c56c2b6e7b1856 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:46.976146 434 raft_consensus.cc:504] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:46.977088 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet db7077d63a534acc94c56c2b6e7b1856 (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.977243 443 ts_tablet_manager.cc:1367] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.249s user 0.060s sys 0.020s
I20240501 14:00:46.977329 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:46.977574 31964 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.978169 443 raft_consensus.cc:348] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.978468 443 raft_consensus.cc:729] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.978746 443 consensus_queue.cc:260] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:46.979360 443 ts_tablet_manager.cc:1398] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.980479 31964 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:46.982688 32032 leader_election.cc:302] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:46.982843 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-60527f81f34b45c199bf52bf0df53ee2 received from {username='slave'} at 127.22.33.69:44499
I20240501 14:00:46.983021 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-60527f81f34b45c199bf52bf0df53ee2 on tablet 60527f81f34b45c199bf52bf0df53ee2 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:46.983033 472 raft_consensus.cc:2781] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.983170 472 raft_consensus.cc:686] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:46.983531 472 consensus_queue.cc:237] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:46.987177 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:46.987453 32228 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.990383 32228 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:46.990547 32726 ts_tablet_manager.cc:1888] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:46.990727 32726 log.cc:1196] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/db7077d63a534acc94c56c2b6e7b1856
I20240501 14:00:46.994539 32726 ts_tablet_manager.cc:1909] T db7077d63a534acc94c56c2b6e7b1856 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:46.994508 471 leader_election.cc:288] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:46.995754 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:46.995962 32361 raft_consensus.cc:3037] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.996129 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet db7077d63a534acc94c56c2b6e7b1856 (table rebalance_test_table_14 [id=7db663aa0244455abdf483182accb2bc]) successfully deleted
I20240501 14:00:46.998833 32361 raft_consensus.cc:2454] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:47.000351 32162 leader_election.cc:302] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:47.000880 432 raft_consensus.cc:2781] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.001071 432 raft_consensus.cc:686] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:47.001399 432 consensus_queue.cc:237] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:47.002161 433 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.004727 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7af304700545b2a28249f9f046cab6" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:47.004959 32096 raft_consensus.cc:3037] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.006845 469 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.007774 32096 raft_consensus.cc:2454] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:47.008404 434 leader_election.cc:288] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:47.011260 466 tablet.cc:1722] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.015849 29103 catalog_manager.cc:5484] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:47.019658 451 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:47.020146 451 tablet_bootstrap.cc:492] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:00:47.020087 29096 catalog_manager.cc:5484] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.021090 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=74688fc9834142a1ae7aa6a2e5bcaac3] alter complete (version 3)
I20240501 14:00:47.023087 451 ts_tablet_manager.cc:1367] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.207s user 0.063s sys 0.000s
I20240501 14:00:47.024012 451 raft_consensus.cc:348] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:47.024268 451 raft_consensus.cc:729] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.024533 451 consensus_queue.cc:260] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:47.025106 451 ts_tablet_manager.cc:1398] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:47.025977 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-49b72e2ac9df4426b071a7e5004e12e1 received from {username='slave'} at 127.22.33.70:36525
I20240501 14:00:47.026167 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-49b72e2ac9df4426b071a7e5004e12e1 on tablet 49b72e2ac9df4426b071a7e5004e12e1 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:47.127703 29635 consensus_queue.cc:237] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:47.128875 432 raft_consensus.cc:482] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.129082 432 raft_consensus.cc:504] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:47.129750 432 leader_election.cc:288] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:47.130636 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:47.130956 31964 raft_consensus.cc:2379] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a446ac802a7457d9f27076781f21d79 in current term 1: Already voted for candidate 0a697f3a1ee544daa7073814ab845d58 in this term.
I20240501 14:00:47.131467 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c84a393bd047d985927417f82bec7d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:47.133152 32163 leader_election.cc:302] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:47.133232 29504 raft_consensus.cc:1261] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.133765 432 raft_consensus.cc:3037] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.138223 29377 raft_consensus.cc:1261] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.139123 408 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.140918 432 raft_consensus.cc:2726] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:47.142699 29096 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:47.142910 29096 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:47.146173 29096 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 8763640f71a3446e9a82f9387ec789f8
I20240501 14:00:47.147027 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet 8763640f71a3446e9a82f9387ec789f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:47.147138 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet 8763640f71a3446e9a82f9387ec789f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:47.147379 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 8763640f71a3446e9a82f9387ec789f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:47.147929 506 tablet_replica.cc:330] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:47.148134 506 raft_consensus.cc:2229] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.148372 506 raft_consensus.cc:2258] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.149371 506 ts_tablet_manager.cc:1875] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.150239 32096 raft_consensus.cc:1203] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:47.156399 506 ts_tablet_manager.cc:1888] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:47.156541 506 log.cc:1196] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/8763640f71a3446e9a82f9387ec789f8
I20240501 14:00:47.157079 506 ts_tablet_manager.cc:1909] T 8763640f71a3446e9a82f9387ec789f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:47.159013 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table_new_name [id=74688fc9834142a1ae7aa6a2e5bcaac3]) successfully deleted
I20240501 14:00:47.162734 501 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.164866 504 tablet_replica.cc:330] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:47.165164 504 raft_consensus.cc:2229] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.165450 504 raft_consensus.cc:2258] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.166039 505 tablet_replica.cc:330] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:47.166265 505 raft_consensus.cc:2229] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:47.166800 505 raft_consensus.cc:2258] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:00:47.166864 29571 consensus_peers.cc:485] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a82cbc0696fe4a8a815eaf0538b5aad8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:47.166973 501 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.167766 504 ts_tablet_manager.cc:1875] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.167560 29504 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.170877 505 ts_tablet_manager.cc:1875] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.173161 29377 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.176920 29103 catalog_manager.cc:5484] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.185292 505 ts_tablet_manager.cc:1888] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:47.185292 504 ts_tablet_manager.cc:1888] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:47.185451 504 log.cc:1196] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/8763640f71a3446e9a82f9387ec789f8
I20240501 14:00:47.185451 505 log.cc:1196] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/8763640f71a3446e9a82f9387ec789f8
I20240501 14:00:47.185917 505 ts_tablet_manager.cc:1909] T 8763640f71a3446e9a82f9387ec789f8 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:47.185917 504 ts_tablet_manager.cc:1909] T 8763640f71a3446e9a82f9387ec789f8 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:47.187062 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table_new_name [id=74688fc9834142a1ae7aa6a2e5bcaac3]) successfully deleted
I20240501 14:00:47.187358 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 8763640f71a3446e9a82f9387ec789f8 (table rebalancer_dynamic_table_new_name [id=74688fc9834142a1ae7aa6a2e5bcaac3]) successfully deleted
I20240501 14:00:47.216122 31964 consensus_queue.cc:237] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:47.219060 32228 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:47.219051 31899 consensus_peers.cc:485] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 414c148db46543b786bfd8b0faecfe96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:47.219147 32096 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:47.219796 526 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:47.220405 526 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:47.224463 32479 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:47.225234 32228 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:47.232585 29105 catalog_manager.cc:5484] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:47.233111 32096 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:47.245251 472 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.249006 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:47.254037 471 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.261145 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]), partition=
I20240501 14:00:47.261534 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.262567 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]), partition=
I20240501 14:00:47.262977 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.269481 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]), partition=
I20240501 14:00:47.269559 545 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:47.269955 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.270354 543 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:47.271616 545 tablet_bootstrap.cc:654] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.272044 543 tablet_bootstrap.cc:654] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.277740 548 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:47.280772 548 tablet_bootstrap.cc:654] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.282169 543 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:47.282402 543 ts_tablet_manager.cc:1367] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240501 14:00:47.283537 543 raft_consensus.cc:348] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.284054 543 raft_consensus.cc:374] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.284206 543 raft_consensus.cc:729] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.284536 543 consensus_queue.cc:260] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.285313 548 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:47.285319 543 ts_tablet_manager.cc:1398] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:47.285526 548 ts_tablet_manager.cc:1367] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240501 14:00:47.288483 545 tablet_bootstrap.cc:492] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:47.288697 545 ts_tablet_manager.cc:1367] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240501 14:00:47.289716 548 raft_consensus.cc:348] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.289743 545 raft_consensus.cc:348] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.289971 545 raft_consensus.cc:374] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.290091 548 raft_consensus.cc:374] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.290124 545 raft_consensus.cc:729] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.290272 548 raft_consensus.cc:729] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.290614 548 consensus_queue.cc:260] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.290436 545 consensus_queue.cc:260] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.291550 545 ts_tablet_manager.cc:1398] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:47.291824 548 ts_tablet_manager.cc:1398] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:47.303020 29635 consensus_queue.cc:237] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:47.305853 32695 raft_consensus.cc:482] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.306038 32695 raft_consensus.cc:504] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.306319 29505 raft_consensus.cc:1261] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.306871 32695 leader_election.cc:288] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:47.307442 501 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:47.307814 29572 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e513634d27c5467fb49c9717c43904cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:47.308162 29377 raft_consensus.cc:1261] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.308650 408 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.322115 408 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:47.324083 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b7cc10aeb4cd1b54932a5ca1f1f32" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:47.324348 29244 raft_consensus.cc:2454] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:47.324915 31898 leader_election.cc:302] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:47.325251 32695 raft_consensus.cc:2781] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.325373 32695 raft_consensus.cc:482] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.325469 32695 raft_consensus.cc:3037] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.327503 29505 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:47.328217 32695 raft_consensus.cc:504] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.328908 32695 leader_election.cc:288] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:47.330374 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b7cc10aeb4cd1b54932a5ca1f1f32" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:47.330610 29244 raft_consensus.cc:3037] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.333612 29244 raft_consensus.cc:2454] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:47.334223 31898 leader_election.cc:302] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:47.334589 32695 raft_consensus.cc:2781] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.335140 32695 raft_consensus.cc:686] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:47.335515 32695 consensus_queue.cc:237] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.336767 29377 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:47.337693 29103 catalog_manager.cc:5484] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:47.338136 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b7cc10aeb4cd1b54932a5ca1f1f32" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:47.338369 29635 raft_consensus.cc:2454] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:47.339205 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b7cc10aeb4cd1b54932a5ca1f1f32" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:47.339468 29635 raft_consensus.cc:3037] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.344286 29635 raft_consensus.cc:2454] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:47.347762 29096 catalog_manager.cc:5484] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:47.399530 29505 consensus_queue.cc:237] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:47.402141 29246 raft_consensus.cc:1261] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.402727 29377 raft_consensus.cc:1261] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.402812 32279 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.403395 32279 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.417995 29096 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:00:47.418327 29096 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:00:47.422688 32279 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.423466 29096 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 9aec4a91f29449f48ac3fcf10d13b6d4
I20240501 14:00:47.424636 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:47.424742 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:47.425160 32725 tablet_replica.cc:330] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:47.425334 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec4a91f29449f48ac3fcf10d13b6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:47.425402 32725 raft_consensus.cc:2229] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.424003 29096 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 040227e018da4e3db04c371c4510b82a
I20240501 14:00:47.425632 32725 raft_consensus.cc:2258] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.426038 29096 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet a61f994801b643e1a369a034580cf3ed
I20240501 14:00:47.426414 32725 ts_tablet_manager.cc:1875] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.426616 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 040227e018da4e3db04c371c4510b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:47.427146 427 tablet_replica.cc:330] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:47.427343 427 raft_consensus.cc:2229] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.427578 427 raft_consensus.cc:2258] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.427717 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 040227e018da4e3db04c371c4510b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:47.428304 427 ts_tablet_manager.cc:1875] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.428870 426 tablet_replica.cc:330] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:47.429095 426 raft_consensus.cc:2229] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:47.429179 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet a61f994801b643e1a369a034580cf3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:47.429442 426 raft_consensus.cc:2258] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.429446 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet a61f994801b643e1a369a034580cf3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:47.429704 32726 tablet_replica.cc:330] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:47.429888 32726 raft_consensus.cc:2229] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:47.430151 426 ts_tablet_manager.cc:1875] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.430215 32726 raft_consensus.cc:2258] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.430945 32726 ts_tablet_manager.cc:1875] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.433163 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 040227e018da4e3db04c371c4510b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:47.434317 29376 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.435424 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet a61f994801b643e1a369a034580cf3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:47.437186 32725 ts_tablet_manager.cc:1888] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.437353 32725 log.cc:1196] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/9aec4a91f29449f48ac3fcf10d13b6d4
I20240501 14:00:47.437820 32725 ts_tablet_manager.cc:1909] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:47.438926 32725 tablet_replica.cc:330] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:47.439082 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.440646 32725 raft_consensus.cc:2229] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.441056 32725 raft_consensus.cc:2258] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:00:47.441447 29441 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a8d8249c87f42faa9a7464369048090. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:47.441776 426 ts_tablet_manager.cc:1888] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.441982 32725 ts_tablet_manager.cc:1875] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.442118 426 log.cc:1196] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/9aec4a91f29449f48ac3fcf10d13b6d4
I20240501 14:00:47.442620 426 ts_tablet_manager.cc:1909] T 9aec4a91f29449f48ac3fcf10d13b6d4 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:47.444152 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.444579 426 tablet_replica.cc:330] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:47.446800 29246 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.447607 29096 catalog_manager.cc:5484] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:47.448137 427 ts_tablet_manager.cc:1888] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.448280 427 log.cc:1196] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/9aec4a91f29449f48ac3fcf10d13b6d4
I20240501 14:00:47.448690 427 ts_tablet_manager.cc:1909] T 9aec4a91f29449f48ac3fcf10d13b6d4 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:47.449581 32726 ts_tablet_manager.cc:1888] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.449720 32726 log.cc:1196] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/a61f994801b643e1a369a034580cf3ed
I20240501 14:00:47.450250 32726 ts_tablet_manager.cc:1909] T a61f994801b643e1a369a034580cf3ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:47.451779 32726 tablet_replica.cc:330] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:47.452231 427 tablet_replica.cc:330] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:47.453541 427 raft_consensus.cc:2229] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.453677 32725 ts_tablet_manager.cc:1888] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.453847 32725 log.cc:1196] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/040227e018da4e3db04c371c4510b82a
I20240501 14:00:47.454118 427 raft_consensus.cc:2258] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.454275 32725 ts_tablet_manager.cc:1909] T 040227e018da4e3db04c371c4510b82a P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:47.455096 32726 raft_consensus.cc:2229] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:47.455160 427 ts_tablet_manager.cc:1875] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.455405 32726 raft_consensus.cc:2258] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.455441 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 040227e018da4e3db04c371c4510b82a (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.455703 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet a61f994801b643e1a369a034580cf3ed (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.455914 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 9aec4a91f29449f48ac3fcf10d13b6d4 (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.456125 32726 ts_tablet_manager.cc:1875] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.458407 426 raft_consensus.cc:2229] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.458832 426 raft_consensus.cc:2258] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.464051 427 ts_tablet_manager.cc:1888] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.465121 427 log.cc:1196] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/a61f994801b643e1a369a034580cf3ed
I20240501 14:00:47.465634 427 ts_tablet_manager.cc:1909] T a61f994801b643e1a369a034580cf3ed P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:47.467092 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet a61f994801b643e1a369a034580cf3ed (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.468487 32726 ts_tablet_manager.cc:1888] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.468634 32726 log.cc:1196] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/040227e018da4e3db04c371c4510b82a
I20240501 14:00:47.468596 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:47.469051 32726 ts_tablet_manager.cc:1909] T 040227e018da4e3db04c371c4510b82a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:47.469951 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 040227e018da4e3db04c371c4510b82a (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.480129 29246 raft_consensus.cc:1261] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:47.480958 29635 raft_consensus.cc:1261] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:47.481781 535 consensus_queue.cc:1035] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.484195 535 consensus_queue.cc:1035] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.487902 432 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.492475 466 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.493058 466 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:47.494978 432 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.504316 29096 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]). Expected version 1 got 0
I20240501 14:00:47.512445 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d] alter complete (version 1)
I20240501 14:00:47.515049 426 ts_tablet_manager.cc:1875] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.544287 29105 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]). Expected version 1 got 0
I20240501 14:00:47.541275 588 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.551640 588 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:47.556514 471 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.564859 426 ts_tablet_manager.cc:1888] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.573516 426 log.cc:1196] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/040227e018da4e3db04c371c4510b82a
I20240501 14:00:47.574937 426 ts_tablet_manager.cc:1909] T 040227e018da4e3db04c371c4510b82a P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:47.575284 31965 raft_consensus.cc:1203] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:47.576522 472 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.576787 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 040227e018da4e3db04c371c4510b82a (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.577275 426 tablet_replica.cc:330] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:47.577646 426 raft_consensus.cc:2229] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.578028 426 raft_consensus.cc:2258] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.579077 426 ts_tablet_manager.cc:1875] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.586644 29096 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]). Expected version 1 got 0
I20240501 14:00:47.588948 598 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.591219 29105 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d]). Expected version 1 got 0
I20240501 14:00:47.593241 598 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:47.597014 426 ts_tablet_manager.cc:1888] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.597326 426 log.cc:1196] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/a61f994801b643e1a369a034580cf3ed
I20240501 14:00:47.597923 426 ts_tablet_manager.cc:1909] T a61f994801b643e1a369a034580cf3ed P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:47.599659 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet a61f994801b643e1a369a034580cf3ed (table rebalance_test_table_16 [id=0d0478d77c2e42ceb6542048eb460da7]) successfully deleted
I20240501 14:00:47.614889 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:47.615515 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:47.626123 466 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.626674 466 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:47.628304 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 1a412c988fd946d7b7d51606be01d1fa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.628633 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 1a412c988fd946d7b7d51606be01d1fa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.628809 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a412c988fd946d7b7d51606be01d1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.629277 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a412c988fd946d7b7d51606be01d1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.630574 588 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.631081 588 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:47.634853 598 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.635404 598 tablet.cc:1620] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:47.636821 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec283b44ec4632b620275af509565e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.641547 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 1a412c988fd946d7b7d51606be01d1fa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.642127 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a412c988fd946d7b7d51606be01d1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.645228 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec283b44ec4632b620275af509565e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.648766 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec283b44ec4632b620275af509565e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.648919 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec283b44ec4632b620275af509565e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.649276 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec283b44ec4632b620275af509565e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.649370 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec283b44ec4632b620275af509565e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.649459 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=9596ea0d6cb74316ad15b211280e388d] alter complete (version 2)
I20240501 14:00:47.657409 543 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:47.658990 543 tablet_bootstrap.cc:654] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.663810 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 8a1690e3a76644859a0711e305b8f946 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.667512 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1690e3a76644859a0711e305b8f946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.669467 548 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:47.671171 548 tablet_bootstrap.cc:654] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.671303 607 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:47.671522 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 8a1690e3a76644859a0711e305b8f946 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.671974 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1690e3a76644859a0711e305b8f946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.672952 607 tablet_bootstrap.cc:654] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.674664 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 8a1690e3a76644859a0711e305b8f946 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.675267 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1690e3a76644859a0711e305b8f946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.678414 543 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:47.678659 543 ts_tablet_manager.cc:1367] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.006s
I20240501 14:00:47.679944 543 raft_consensus.cc:348] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.680303 543 raft_consensus.cc:374] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.680445 543 raft_consensus.cc:729] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.680771 543 consensus_queue.cc:260] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.681512 543 ts_tablet_manager.cc:1398] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:47.696218 543 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:47.698802 543 tablet_bootstrap.cc:654] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.700748 611 ts_tablet_manager.cc:902] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:47.703760 548 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:47.704089 609 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:47.706539 609 tablet_bootstrap.cc:654] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.706887 548 ts_tablet_manager.cc:1367] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.005s
I20240501 14:00:47.708077 548 raft_consensus.cc:348] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.709369 605 ts_tablet_manager.cc:902] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:47.709427 548 raft_consensus.cc:374] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.709654 548 raft_consensus.cc:729] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.710215 548 consensus_queue.cc:260] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.711280 545 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:47.712116 543 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:47.712273 543 ts_tablet_manager.cc:1367] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240501 14:00:47.712832 545 tablet_bootstrap.cc:654] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.713188 543 raft_consensus.cc:348] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.713424 543 raft_consensus.cc:374] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.713513 543 raft_consensus.cc:729] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.713771 543 consensus_queue.cc:260] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.714342 543 ts_tablet_manager.cc:1398] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:47.711091 607 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:47.717387 607 ts_tablet_manager.cc:1367] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.020s
I20240501 14:00:47.718664 607 raft_consensus.cc:348] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.720270 607 raft_consensus.cc:374] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.720494 607 raft_consensus.cc:729] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.721124 607 consensus_queue.cc:260] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.727072 548 ts_tablet_manager.cc:1398] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240501 14:00:47.729648 607 ts_tablet_manager.cc:1398] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:00:47.730161 32602 raft_consensus.cc:482] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.730361 32602 raft_consensus.cc:504] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.731429 32602 leader_election.cc:288] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:47.737917 609 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:47.738273 609 ts_tablet_manager.cc:1367] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20240501 14:00:47.739426 609 raft_consensus.cc:348] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.741194 609 raft_consensus.cc:374] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.741606 609 raft_consensus.cc:729] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.742174 609 consensus_queue.cc:260] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.745008 609 ts_tablet_manager.cc:1398] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:47.749877 607 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:47.750701 622 ts_tablet_manager.cc:902] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:47.754482 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:47.766440 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a412c988fd946d7b7d51606be01d1fa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:47.766961 32097 raft_consensus.cc:2454] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:47.767696 29181 leader_election.cc:302] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:47.768607 32602 raft_consensus.cc:2781] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.768730 32602 raft_consensus.cc:482] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.768841 32602 raft_consensus.cc:3037] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.771775 32602 raft_consensus.cc:504] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.774684 616 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:47.776499 616 tablet_bootstrap.cc:654] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.777577 32602 leader_election.cc:288] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:47.778108 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a412c988fd946d7b7d51606be01d1fa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:47.778321 466 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.778432 32097 raft_consensus.cc:3037] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.779780 598 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.782554 32097 raft_consensus.cc:2454] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:47.783238 29181 leader_election.cc:302] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:47.783761 32602 raft_consensus.cc:2781] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.784034 32602 raft_consensus.cc:686] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:47.784500 32602 consensus_queue.cc:237] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:47.786406 545 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:47.786660 545 ts_tablet_manager.cc:1367] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.003s
I20240501 14:00:47.786804 622 tablet_copy_client.cc:308] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:00:47.787684 607 tablet_bootstrap.cc:654] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.794763 588 tablet.cc:1722] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:47.796288 607 tablet_bootstrap.cc:492] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:47.796602 607 ts_tablet_manager.cc:1367] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.004s
I20240501 14:00:47.797878 607 raft_consensus.cc:348] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.798135 607 raft_consensus.cc:374] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.798225 607 raft_consensus.cc:729] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.798507 607 consensus_queue.cc:260] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.801728 607 ts_tablet_manager.cc:1398] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:47.802403 607 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:47.804992 607 tablet_bootstrap.cc:654] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.806102 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=9596ea0d6cb74316ad15b211280e388d] alter complete (version 3)
I20240501 14:00:47.810484 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet e513634d27c5467fb49c9717c43904cd from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:48851)
I20240501 14:00:47.810750 607 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:47.810741 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet e513634d27c5467fb49c9717c43904cd from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:48851: session id = 0a697f3a1ee544daa7073814ab845d58-e513634d27c5467fb49c9717c43904cd
I20240501 14:00:47.811082 607 ts_tablet_manager.cc:1367] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240501 14:00:47.812192 616 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:47.812289 607 raft_consensus.cc:348] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.812587 607 raft_consensus.cc:374] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.812696 607 raft_consensus.cc:729] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.813014 616 ts_tablet_manager.cc:1367] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240501 14:00:47.813012 607 consensus_queue.cc:260] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.813819 607 ts_tablet_manager.cc:1398] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:47.816093 29657 tablet_copy_source_session.cc:205] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:47.818598 29096 catalog_manager.cc:5484] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:47.822901 616 raft_consensus.cc:348] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.823861 616 raft_consensus.cc:374] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.824187 616 raft_consensus.cc:729] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.824739 616 consensus_queue.cc:260] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.825043 622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e513634d27c5467fb49c9717c43904cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.825762 616 ts_tablet_manager.cc:1398] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:47.829540 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a412c988fd946d7b7d51606be01d1fa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:47.829841 32362 raft_consensus.cc:2454] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:47.830379 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a412c988fd946d7b7d51606be01d1fa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:47.830633 32361 raft_consensus.cc:3037] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.837460 32361 raft_consensus.cc:2454] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:47.843320 545 raft_consensus.cc:348] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.844194 545 raft_consensus.cc:374] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.844861 545 raft_consensus.cc:729] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.845136 622 tablet_copy_client.cc:783] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:47.845580 545 consensus_queue.cc:260] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:47.852265 622 tablet_copy_client.cc:647] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:47.853049 545 ts_tablet_manager.cc:1398] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.066s user 0.000s sys 0.002s
I20240501 14:00:47.855566 622 tablet_copy_client.cc:523] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:47.856458 29245 consensus_queue.cc:237] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.859302 622 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:47.859282 611 tablet_copy_client.cc:308] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:00:47.860165 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet 414c148db46543b786bfd8b0faecfe96 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:34297)
I20240501 14:00:47.860388 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet 414c148db46543b786bfd8b0faecfe96 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:34297: session id = 6edd6a3430d44c38a19c2804e555059f-414c148db46543b786bfd8b0faecfe96
W20240501 14:00:47.861567 29181 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b925d954c9143279ba69ae5c90d3266. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:47.862569 29635 raft_consensus.cc:1261] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.863214 514 raft_consensus.cc:482] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.863353 514 raft_consensus.cc:504] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.863961 31985 tablet_copy_source_session.cc:205] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:47.864094 514 leader_election.cc:288] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:47.864264 514 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.865211 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec283b44ec4632b620275af509565e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:47.865455 31964 raft_consensus.cc:2454] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:47.866115 29181 leader_election.cc:302] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:47.866194 611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414c148db46543b786bfd8b0faecfe96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.866912 32602 raft_consensus.cc:2781] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.867012 32602 raft_consensus.cc:482] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.867110 32602 raft_consensus.cc:3037] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.869838 32602 raft_consensus.cc:504] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.871830 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec283b44ec4632b620275af509565e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:47.872049 31964 raft_consensus.cc:3037] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.873160 611 tablet_copy_client.cc:783] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:47.873411 611 tablet_copy_client.cc:647] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:47.874886 31964 raft_consensus.cc:2454] T 83ec283b44ec4632b620275af509565e P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:47.875388 611 tablet_copy_client.cc:523] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:47.875546 29181 leader_election.cc:302] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:47.876008 514 raft_consensus.cc:2781] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.879426 611 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:47.881426 32602 leader_election.cc:288] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:47.884688 29505 raft_consensus.cc:1261] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.886054 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec283b44ec4632b620275af509565e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:47.886292 32361 raft_consensus.cc:2454] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:47.886615 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec283b44ec4632b620275af509565e" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:47.886768 32362 raft_consensus.cc:3037] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.888967 32602 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.890096 32362 raft_consensus.cc:2454] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:47.895861 611 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:47.896482 611 tablet_bootstrap.cc:492] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:00:47.896811 611 ts_tablet_manager.cc:1367] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20240501 14:00:47.898000 611 raft_consensus.cc:348] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:47.898222 611 raft_consensus.cc:729] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.898857 611 consensus_queue.cc:260] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:47.899713 611 ts_tablet_manager.cc:1398] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:00:47.900730 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-414c148db46543b786bfd8b0faecfe96 received from {username='slave'} at 127.22.33.72:34297
I20240501 14:00:47.900949 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-414c148db46543b786bfd8b0faecfe96 on tablet 414c148db46543b786bfd8b0faecfe96 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:47.902554 644 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:47.905367 29505 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:47.908799 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:47.910080 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:47.915043 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 986b7cc10aeb4cd1b54932a5ca1f1f32
I20240501 14:00:47.916081 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:47.920529 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:47.920666 649 tablet_replica.cc:330] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:47.920895 32726 tablet_replica.cc:330] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:47.921116 32726 raft_consensus.cc:2229] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.921355 32726 raft_consensus.cc:2258] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.922415 32726 ts_tablet_manager.cc:1875] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.920922 649 raft_consensus.cc:2229] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:47.924811 649 raft_consensus.cc:2258] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.925858 649 ts_tablet_manager.cc:1875] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.933439 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:00:47.933779 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:00:47.940881 29096 catalog_manager.cc:5484] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:47.943354 32726 ts_tablet_manager.cc:1888] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:47.945408 32726 log.cc:1196] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/986b7cc10aeb4cd1b54932a5ca1f1f32
I20240501 14:00:47.946163 32726 ts_tablet_manager.cc:1909] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:47.947919 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table_new_name [id=9596ea0d6cb74316ad15b211280e388d]) successfully deleted
I20240501 14:00:47.971778 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:47.928052 514 raft_consensus.cc:686] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:47.974031 514 consensus_queue.cc:237] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:47.984231 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 37988c87ab494486be42cc398f307830
I20240501 14:00:47.985325 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 37988c87ab494486be42cc398f307830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:47.985594 426 tablet_replica.cc:330] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:47.985803 426 raft_consensus.cc:2229] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.986006 426 raft_consensus.cc:2258] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.986737 426 ts_tablet_manager.cc:1875] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:47.992712 653 tablet_replica.cc:330] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:47.993163 653 raft_consensus.cc:2229] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.993580 653 raft_consensus.cc:2258] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.994832 426 ts_tablet_manager.cc:1888] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:47.994999 426 log.cc:1196] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/37988c87ab494486be42cc398f307830
I20240501 14:00:47.995440 426 ts_tablet_manager.cc:1909] T 37988c87ab494486be42cc398f307830 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:47.996820 650 ts_tablet_manager.cc:902] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:47.997228 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 37988c87ab494486be42cc398f307830 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:47.998821 649 ts_tablet_manager.cc:1888] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:47.999035 649 log.cc:1196] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/986b7cc10aeb4cd1b54932a5ca1f1f32
I20240501 14:00:47.999554 649 ts_tablet_manager.cc:1909] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:48.000949 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table_new_name [id=9596ea0d6cb74316ad15b211280e388d]) successfully deleted
I20240501 14:00:48.003034 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 37988c87ab494486be42cc398f307830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:48.003391 32726 tablet_replica.cc:330] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:48.003683 32726 raft_consensus.cc:2229] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:48.003829 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet bb1805e53a764f8db7d88a19b585bccf
I20240501 14:00:48.004041 32726 raft_consensus.cc:2258] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.004720 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1805e53a764f8db7d88a19b585bccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:48.005380 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1805e53a764f8db7d88a19b585bccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:48.005702 32726 ts_tablet_manager.cc:1875] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.005966 426 tablet_replica.cc:330] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:48.006167 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 37988c87ab494486be42cc398f307830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:48910
W20240501 14:00:48.006320 29181 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 49034.9us
I20240501 14:00:48.006393 605 tablet_copy_client.cc:308] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:00:48.007247 426 raft_consensus.cc:2229] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.007629 426 raft_consensus.cc:2258] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.009732 426 ts_tablet_manager.cc:1875] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.011529 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 623fd5c168d34a93a3dd77249aa9a391
I20240501 14:00:48.012060 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1805e53a764f8db7d88a19b585bccf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:48.012363 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:48.024178 426 ts_tablet_manager.cc:1888] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.024523 32726 ts_tablet_manager.cc:1888] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.024755 32726 log.cc:1196] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/37988c87ab494486be42cc398f307830
I20240501 14:00:48.026144 32726 ts_tablet_manager.cc:1909] T 37988c87ab494486be42cc398f307830 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:48.027869 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 37988c87ab494486be42cc398f307830 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.011291 653 ts_tablet_manager.cc:1875] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.030033 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:48.035326 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:48.038605 29637 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:48.042444 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet a82cbc0696fe4a8a815eaf0538b5aad8 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:59115)
I20240501 14:00:48.043926 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]), partition=
I20240501 14:00:48.044342 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7ab679e7324e70a1a9ac18fe38ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.044584 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]), partition=
I20240501 14:00:48.046036 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7ab679e7324e70a1a9ac18fe38ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.049060 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 623fd5c168d34a93a3dd77249aa9a391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:47 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:48.051492 607 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:48.051584 29105 catalog_manager.cc:5484] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:48.053107 607 tablet_bootstrap.cc:654] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.054071 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]), partition=
I20240501 14:00:48.054603 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7ab679e7324e70a1a9ac18fe38ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.059055 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet a82cbc0696fe4a8a815eaf0538b5aad8 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:59115: session id = 6a446ac802a7457d9f27076781f21d79-a82cbc0696fe4a8a815eaf0538b5aad8
I20240501 14:00:48.059612 657 tablet_replica.cc:330] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:48.059947 657 raft_consensus.cc:2229] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:48.060390 657 raft_consensus.cc:2258] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.062362 29657 tablet_copy_source_session.cc:205] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.062861 657 ts_tablet_manager.cc:1875] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.064512 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82cbc0696fe4a8a815eaf0538b5aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.068650 653 ts_tablet_manager.cc:1888] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:48.068773 653 log.cc:1196] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/986b7cc10aeb4cd1b54932a5ca1f1f32
I20240501 14:00:48.070626 657 ts_tablet_manager.cc:1888] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.070816 657 log.cc:1196] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb1805e53a764f8db7d88a19b585bccf
I20240501 14:00:48.071269 657 ts_tablet_manager.cc:1909] T bb1805e53a764f8db7d88a19b585bccf P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:48.071933 605 tablet_copy_client.cc:783] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.072403 605 tablet_copy_client.cc:647] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.073478 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet bb1805e53a764f8db7d88a19b585bccf (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.075004 605 tablet_copy_client.cc:523] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.078231 501 raft_consensus.cc:482] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.078449 501 raft_consensus.cc:504] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.079159 501 leader_election.cc:288] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:48.079978 605 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:48.080349 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a1690e3a76644859a0711e305b8f946" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:48.080593 32362 raft_consensus.cc:2454] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:48.080964 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a1690e3a76644859a0711e305b8f946" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:48.081203 32227 raft_consensus.cc:2454] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:48.081975 29571 leader_election.cc:302] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:48.082535 501 raft_consensus.cc:2781] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.082628 501 raft_consensus.cc:482] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.082734 501 raft_consensus.cc:3037] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.085286 609 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:48.086112 501 raft_consensus.cc:504] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.087081 609 tablet_bootstrap.cc:654] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.087558 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a1690e3a76644859a0711e305b8f946" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:48.087790 32362 raft_consensus.cc:3037] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.090565 32362 raft_consensus.cc:2454] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:48.091444 29571 leader_election.cc:302] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:48.091856 501 leader_election.cc:288] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:48.092068 501 raft_consensus.cc:2781] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.092255 501 raft_consensus.cc:686] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:48.092584 501 consensus_queue.cc:237] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.098224 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a1690e3a76644859a0711e305b8f946" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:48.098495 32227 raft_consensus.cc:3037] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.101329 616 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:48.103132 616 tablet_bootstrap.cc:654] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.104911 32227 raft_consensus.cc:2454] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:48.107296 29103 catalog_manager.cc:5484] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:48.109112 426 log.cc:1196] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb1805e53a764f8db7d88a19b585bccf
I20240501 14:00:48.109668 426 ts_tablet_manager.cc:1909] T bb1805e53a764f8db7d88a19b585bccf P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:48.110646 426 tablet_replica.cc:330] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:48.110839 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet bb1805e53a764f8db7d88a19b585bccf (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.110906 426 raft_consensus.cc:2229] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.111151 426 raft_consensus.cc:2258] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.111923 426 ts_tablet_manager.cc:1875] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.113204 32726 tablet_replica.cc:330] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:48.113441 32726 raft_consensus.cc:2229] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.113735 32726 raft_consensus.cc:2258] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.114605 32726 ts_tablet_manager.cc:1875] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.117344 653 ts_tablet_manager.cc:1909] T 986b7cc10aeb4cd1b54932a5ca1f1f32 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:48.118381 616 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:48.118642 616 ts_tablet_manager.cc:1367] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240501 14:00:48.119624 616 raft_consensus.cc:348] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.119854 607 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:48.120023 616 raft_consensus.cc:374] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.120059 607 ts_tablet_manager.cc:1367] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.003s
I20240501 14:00:48.120183 616 raft_consensus.cc:729] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.120527 616 consensus_queue.cc:260] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.121034 607 raft_consensus.cc:348] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.121306 607 raft_consensus.cc:374] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.121374 607 raft_consensus.cc:729] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.121670 607 consensus_queue.cc:260] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.122328 616 ts_tablet_manager.cc:1398] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:48.123211 607 ts_tablet_manager.cc:1398] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:48.125618 653 tablet_replica.cc:330] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:48.125897 653 raft_consensus.cc:2229] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 14:00:48.126056 29181 consensus_peers.cc:485] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.126140 653 raft_consensus.cc:2258] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.126291 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 986b7cc10aeb4cd1b54932a5ca1f1f32 (table rebalancer_dynamic_table_new_name [id=9596ea0d6cb74316ad15b211280e388d]) successfully deleted
I20240501 14:00:48.127836 653 ts_tablet_manager.cc:1875] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.128528 609 tablet_bootstrap.cc:492] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:48.129891 609 ts_tablet_manager.cc:1367] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.005s
I20240501 14:00:48.130743 609 raft_consensus.cc:348] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.131120 609 raft_consensus.cc:374] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.131229 609 raft_consensus.cc:729] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.131529 609 consensus_queue.cc:260] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.133131 609 ts_tablet_manager.cc:1398] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:48.141263 650 tablet_copy_client.cc:308] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:00:48.155107 32726 ts_tablet_manager.cc:1888] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.163933 426 ts_tablet_manager.cc:1888] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.169134 32726 log.cc:1196] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb1805e53a764f8db7d88a19b585bccf
I20240501 14:00:48.169745 32726 ts_tablet_manager.cc:1909] T bb1805e53a764f8db7d88a19b585bccf P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:48.175024 426 log.cc:1196] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/623fd5c168d34a93a3dd77249aa9a391
I20240501 14:00:48.177055 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 5a8d8249c87f42faa9a7464369048090 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:41457)
I20240501 14:00:48.177248 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 5a8d8249c87f42faa9a7464369048090 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:41457: session id = 6a446ac802a7457d9f27076781f21d79-5a8d8249c87f42faa9a7464369048090
I20240501 14:00:48.180088 29527 tablet_copy_source_session.cc:205] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.181782 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8249c87f42faa9a7464369048090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.182157 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet bb1805e53a764f8db7d88a19b585bccf (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.182530 32726 tablet_replica.cc:330] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:48.182933 426 ts_tablet_manager.cc:1909] T 623fd5c168d34a93a3dd77249aa9a391 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:48.184000 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 623fd5c168d34a93a3dd77249aa9a391 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.184348 32726 raft_consensus.cc:2229] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.184593 32726 raft_consensus.cc:2258] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.185366 32726 ts_tablet_manager.cc:1875] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.189575 650 tablet_copy_client.cc:783] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.189973 650 tablet_copy_client.cc:647] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.191232 653 ts_tablet_manager.cc:1888] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.191469 653 log.cc:1196] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/37988c87ab494486be42cc398f307830
I20240501 14:00:48.192909 653 ts_tablet_manager.cc:1909] T 37988c87ab494486be42cc398f307830 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:48.194036 653 tablet_replica.cc:330] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:48.194120 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 37988c87ab494486be42cc398f307830 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.194396 653 raft_consensus.cc:2229] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:48.194793 653 raft_consensus.cc:2258] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.195463 622 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:48.195580 653 ts_tablet_manager.cc:1875] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.196075 622 tablet_bootstrap.cc:492] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:00:48.196411 622 ts_tablet_manager.cc:1367] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.337s user 0.061s sys 0.033s
I20240501 14:00:48.196890 650 tablet_copy_client.cc:523] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.197412 622 raft_consensus.cc:348] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.200160 622 raft_consensus.cc:729] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.200831 32362 raft_consensus.cc:1203] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:48.200560 622 consensus_queue.cc:260] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.207016 622 ts_tablet_manager.cc:1398] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:00:48.208420 471 raft_consensus.cc:482] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.208600 471 raft_consensus.cc:504] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.210218 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7ab679e7324e70a1a9ac18fe38ad05" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:48.210624 32227 raft_consensus.cc:2454] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:48.210582 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7ab679e7324e70a1a9ac18fe38ad05" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:48.210846 32362 raft_consensus.cc:2454] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:48.212311 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-e513634d27c5467fb49c9717c43904cd received from {username='slave'} at 127.22.33.70:48851
I20240501 14:00:48.212488 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-e513634d27c5467fb49c9717c43904cd on tablet e513634d27c5467fb49c9717c43904cd with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:48.212208 650 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:48.215739 653 ts_tablet_manager.cc:1888] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.205907 32726 ts_tablet_manager.cc:1888] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:48.219007 471 leader_election.cc:288] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:48.219233 32726 log.cc:1196] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/623fd5c168d34a93a3dd77249aa9a391
I20240501 14:00:48.219686 32031 leader_election.cc:302] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:48.220307 471 raft_consensus.cc:2781] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.220432 471 raft_consensus.cc:482] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.220525 471 raft_consensus.cc:3037] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.220872 32726 ts_tablet_manager.cc:1909] T 623fd5c168d34a93a3dd77249aa9a391 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:48.221908 653 log.cc:1196] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/623fd5c168d34a93a3dd77249aa9a391
I20240501 14:00:48.222378 653 ts_tablet_manager.cc:1909] T 623fd5c168d34a93a3dd77249aa9a391 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:48.223275 471 raft_consensus.cc:504] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.224192 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 623fd5c168d34a93a3dd77249aa9a391 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.224480 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7ab679e7324e70a1a9ac18fe38ad05" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:48.224725 32227 raft_consensus.cc:3037] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.224752 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 623fd5c168d34a93a3dd77249aa9a391 (table rebalance_test_table_18 [id=afecfebb80344e81b07b7d25a8ba0025]) successfully deleted
I20240501 14:00:48.225474 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7ab679e7324e70a1a9ac18fe38ad05" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:48.225718 32362 raft_consensus.cc:3037] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.227705 32227 raft_consensus.cc:2454] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:48.228227 32031 leader_election.cc:302] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:48.228288 32362 raft_consensus.cc:2454] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:48.231591 670 raft_consensus.cc:2781] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.234967 471 leader_election.cc:288] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:48.261428 670 raft_consensus.cc:686] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:48.262140 670 consensus_queue.cc:237] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:48.274734 29101 catalog_manager.cc:5484] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:48.326776 605 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:48.327490 605 tablet_bootstrap.cc:492] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:00:48.327921 605 ts_tablet_manager.cc:1367] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.248s user 0.051s sys 0.017s
I20240501 14:00:48.328925 605 raft_consensus.cc:348] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:48.329510 605 raft_consensus.cc:729] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.329923 605 consensus_queue.cc:260] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:48.331538 605 ts_tablet_manager.cc:1398] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:48.331794 650 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:48.332278 650 tablet_bootstrap.cc:492] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:00:48.332397 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-a82cbc0696fe4a8a815eaf0538b5aad8 received from {username='slave'} at 127.22.33.71:59115
I20240501 14:00:48.332540 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-a82cbc0696fe4a8a815eaf0538b5aad8 on tablet a82cbc0696fe4a8a815eaf0538b5aad8 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:48.332598 650 ts_tablet_manager.cc:1367] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.120s user 0.068s sys 0.033s
I20240501 14:00:48.333595 650 raft_consensus.cc:348] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:48.334436 650 raft_consensus.cc:729] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.334798 650 consensus_queue.cc:260] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:48.335950 650 ts_tablet_manager.cc:1398] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:48.336748 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-5a8d8249c87f42faa9a7464369048090 received from {username='slave'} at 127.22.33.71:41457
I20240501 14:00:48.336927 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-5a8d8249c87f42faa9a7464369048090 on tablet 5a8d8249c87f42faa9a7464369048090 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:48.350360 32097 raft_consensus.cc:1203] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:48.351727 29245 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:48.354331 29377 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.355051 652 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.355266 29505 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:48.356047 29180 consensus_peers.cc:485] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 578c3f4c5a7c4e2c962b64736195f0bc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.356554 652 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.366470 651 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:48.367138 29505 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:48.373759 29101 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:48.377326 29377 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:48.386761 693 ts_tablet_manager.cc:902] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:48.388288 652 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.395143 651 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:00:48.411094 652 consensus_queue.cc:1035] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.412554 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:48.423962 652 consensus_queue.cc:1035] T 83ec283b44ec4632b620275af509565e P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.487031 693 tablet_copy_client.cc:308] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:00:48.488224 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 8b925d954c9143279ba69ae5c90d3266 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:44499)
I20240501 14:00:48.488468 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 8b925d954c9143279ba69ae5c90d3266 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:44499: session id = 9903ed35bca144bcb74e9bf5102117e7-8b925d954c9143279ba69ae5c90d3266
I20240501 14:00:48.491402 29266 tablet_copy_source_session.cc:205] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.494822 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b925d954c9143279ba69ae5c90d3266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.507277 32362 raft_consensus.cc:1261] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.508105 670 consensus_queue.cc:1035] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.516499 32227 raft_consensus.cc:1261] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.517309 670 consensus_queue.cc:1035] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.526960 29105 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]). Expected version 1 got 0
I20240501 14:00:48.535982 408 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.557572 717 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.560489 29105 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]). Expected version 1 got 0
I20240501 14:00:48.566416 408 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.573352 717 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:48.574124 716 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.574776 716 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:48.575320 693 tablet_copy_client.cc:783] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.575708 693 tablet_copy_client.cc:647] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.577848 693 tablet_copy_client.cc:523] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.583334 29105 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]). Expected version 1 got 0
I20240501 14:00:48.593394 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1] alter complete (version 1)
I20240501 14:00:48.594971 715 mvcc.cc:205] Tried to move back new op lower bound from 7022887110619885568 to 7022887109706235904. Current Snapshot: MvccSnapshot[applied={T|T < 7022887110619885568}]
I20240501 14:00:48.599891 725 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.600899 725 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:48.602892 693 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:48.606274 29105 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1]). Expected version 1 got 0
I20240501 14:00:48.629977 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:48.643884 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb60619c1a4d8e8cd6266596a0614d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.643878 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb60619c1a4d8e8cd6266596a0614d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.643901 29356 tablet_service.cc:1467] Processing CreateTablet for tablet e41986f1ae7648a5bfe7760c069af672 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.644361 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb60619c1a4d8e8cd6266596a0614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.644384 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb60619c1a4d8e8cd6266596a0614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.644403 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41986f1ae7648a5bfe7760c069af672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.649735 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbb60619c1a4d8e8cd6266596a0614d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.650182 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb60619c1a4d8e8cd6266596a0614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.651108 31945 tablet_service.cc:1467] Processing CreateTablet for tablet e41986f1ae7648a5bfe7760c069af672 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.651551 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41986f1ae7648a5bfe7760c069af672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.653595 29224 tablet_service.cc:1467] Processing CreateTablet for tablet e41986f1ae7648a5bfe7760c069af672 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.653755 32077 tablet_service.cc:1467] Processing CreateTablet for tablet fee3fe52d2ae462a823d3ac03f2ce8bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.654017 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41986f1ae7648a5bfe7760c069af672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.654207 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3fe52d2ae462a823d3ac03f2ce8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.654670 32207 tablet_service.cc:1467] Processing CreateTablet for tablet fee3fe52d2ae462a823d3ac03f2ce8bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.655064 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3fe52d2ae462a823d3ac03f2ce8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.662904 29615 tablet_service.cc:1467] Processing CreateTablet for tablet fee3fe52d2ae462a823d3ac03f2ce8bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.663347 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3fe52d2ae462a823d3ac03f2ce8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.679186 731 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:48.680227 734 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:48.681706 731 tablet_bootstrap.cc:654] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.682909 734 tablet_bootstrap.cc:654] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.686143 730 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:48.689493 730 tablet_bootstrap.cc:654] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.691226 732 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:48.692067 736 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:48.692725 732 tablet_bootstrap.cc:654] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.694695 735 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:48.695271 734 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:48.695492 734 ts_tablet_manager.cc:1367] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 14:00:48.695883 731 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:48.696404 734 raft_consensus.cc:348] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.697206 734 raft_consensus.cc:374] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.697332 731 ts_tablet_manager.cc:1367] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240501 14:00:48.697355 734 raft_consensus.cc:729] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.697780 734 consensus_queue.cc:260] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.697997 736 tablet_bootstrap.cc:654] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.698571 733 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:48.699913 734 ts_tablet_manager.cc:1398] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:48.700031 731 raft_consensus.cc:348] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.700248 734 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:48.700258 731 raft_consensus.cc:374] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.700348 731 raft_consensus.cc:729] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.701449 731 consensus_queue.cc:260] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.702725 735 tablet_bootstrap.cc:654] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.702953 734 tablet_bootstrap.cc:654] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.703727 731 ts_tablet_manager.cc:1398] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:48.709183 730 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:48.709370 730 ts_tablet_manager.cc:1367] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240501 14:00:48.710280 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:48.712371 730 raft_consensus.cc:348] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.712606 730 raft_consensus.cc:374] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.712716 730 raft_consensus.cc:729] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.712970 730 consensus_queue.cc:260] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.713558 730 ts_tablet_manager.cc:1398] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:48.714061 734 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:48.714304 734 ts_tablet_manager.cc:1367] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240501 14:00:48.719400 734 raft_consensus.cc:348] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.719759 734 raft_consensus.cc:374] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.719887 734 raft_consensus.cc:729] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.720182 734 consensus_queue.cc:260] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.722771 734 ts_tablet_manager.cc:1398] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:48.725724 716 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:48.726253 716 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:48.728835 725 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:48.729486 725 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:48.731884 732 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:48.732175 732 ts_tablet_manager.cc:1367] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.010s
I20240501 14:00:48.733295 732 raft_consensus.cc:348] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.733642 732 raft_consensus.cc:374] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.736688 732 raft_consensus.cc:729] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.737112 732 consensus_queue.cc:260] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.738986 717 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:48.739668 717 tablet.cc:1620] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:48.741030 733 tablet_bootstrap.cc:654] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.748323 32227 raft_consensus.cc:1203] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:48.753160 732 ts_tablet_manager.cc:1398] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:00:48.754073 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=914d070f147e41298d56fcfaf213b5c1] alter complete (version 2)
I20240501 14:00:48.755234 736 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:48.755481 736 ts_tablet_manager.cc:1367] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.008s
I20240501 14:00:48.757143 736 raft_consensus.cc:348] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.758260 735 tablet_bootstrap.cc:492] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:48.758508 735 ts_tablet_manager.cc:1367] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.000s
I20240501 14:00:48.758697 736 raft_consensus.cc:374] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.759871 735 raft_consensus.cc:348] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.760263 736 raft_consensus.cc:729] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.760721 736 consensus_queue.cc:260] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.761235 735 raft_consensus.cc:374] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.761440 735 raft_consensus.cc:729] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.761857 735 consensus_queue.cc:260] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.762867 735 ts_tablet_manager.cc:1398] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:48.767956 736 ts_tablet_manager.cc:1398] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20240501 14:00:48.774747 736 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:48.775828 31964 consensus_queue.cc:237] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:48.776540 736 tablet_bootstrap.cc:654] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.776615 32227 raft_consensus.cc:1203] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:00:48.789049 31900 consensus_peers.cc:485] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c86d880c077411a99e6b9597adeeb0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.790355 32097 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.790433 733 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:48.791049 733 ts_tablet_manager.cc:1367] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.015s
I20240501 14:00:48.791164 32695 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:48.792213 29244 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.792549 733 raft_consensus.cc:348] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.792769 733 raft_consensus.cc:374] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.792886 733 raft_consensus.cc:729] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.793138 733 consensus_queue.cc:260] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.794062 32695 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:48.794562 736 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:48.794736 736 ts_tablet_manager.cc:1367] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.003s
I20240501 14:00:48.795529 736 raft_consensus.cc:348] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.795753 736 raft_consensus.cc:374] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.795836 736 raft_consensus.cc:729] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.795902 733 ts_tablet_manager.cc:1398] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:48.796082 736 consensus_queue.cc:260] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.796569 736 ts_tablet_manager.cc:1398] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:48.802626 693 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:48.803125 693 tablet_bootstrap.cc:492] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:00:48.803397 675 raft_consensus.cc:482] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.803481 693 ts_tablet_manager.cc:1367] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.201s user 0.042s sys 0.015s
I20240501 14:00:48.803575 675 raft_consensus.cc:504] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.804265 693 raft_consensus.cc:348] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.804487 693 raft_consensus.cc:729] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.804517 675 leader_election.cc:288] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:48.804780 693 consensus_queue.cc:260] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:48.805780 693 ts_tablet_manager.cc:1398] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:48.806321 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee3fe52d2ae462a823d3ac03f2ce8bc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:48.806604 29637 raft_consensus.cc:2454] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:48.807641 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-8b925d954c9143279ba69ae5c90d3266 received from {username='slave'} at 127.22.33.69:44499
I20240501 14:00:48.807799 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-8b925d954c9143279ba69ae5c90d3266 on tablet 8b925d954c9143279ba69ae5c90d3266 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:48.808424 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee3fe52d2ae462a823d3ac03f2ce8bc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:48.808714 32227 raft_consensus.cc:2454] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:48.809365 32030 leader_election.cc:302] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:48.809759 675 raft_consensus.cc:2781] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.809904 675 raft_consensus.cc:482] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.810000 675 raft_consensus.cc:3037] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.816493 753 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:48.817322 29244 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:48.830300 32097 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:48.836670 675 raft_consensus.cc:504] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.838021 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee3fe52d2ae462a823d3ac03f2ce8bc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:48.838316 32227 raft_consensus.cc:3037] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.840513 29105 catalog_manager.cc:5484] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:48.842856 675 leader_election.cc:288] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:48.844097 32227 raft_consensus.cc:2454] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:48.845007 32031 leader_election.cc:302] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:48.845471 675 raft_consensus.cc:2781] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.845744 675 raft_consensus.cc:686] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:48.846040 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee3fe52d2ae462a823d3ac03f2ce8bc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:48.846334 29637 raft_consensus.cc:3037] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.846061 675 consensus_queue.cc:237] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.860751 31964 raft_consensus.cc:1203] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:48.862721 29637 raft_consensus.cc:2454] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:48.864733 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:48.886013 716 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.888031 725 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.891911 717 tablet.cc:1722] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.906760 589 raft_consensus.cc:482] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.906945 589 raft_consensus.cc:504] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.907739 589 leader_election.cc:288] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:48.908665 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:48.908901 32360 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:48.909215 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=914d070f147e41298d56fcfaf213b5c1] alter complete (version 3)
I20240501 14:00:48.909565 32162 leader_election.cc:302] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:48.910017 589 raft_consensus.cc:2781] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.910115 589 raft_consensus.cc:482] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.910182 589 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.911885 765 ts_tablet_manager.cc:902] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:48.913820 589 raft_consensus.cc:504] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.915117 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:48.915336 32362 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.918663 32362 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:48.919801 589 leader_election.cc:288] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:48.920337 32162 leader_election.cc:302] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:48.920684 589 raft_consensus.cc:2781] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.920883 589 raft_consensus.cc:686] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:48.921229 589 consensus_queue.cc:237] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:48.921741 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:48.922020 29635 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.934813 29635 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:48.939143 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:48.939514 29637 raft_consensus.cc:2362] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1. Re-sending same reply.
I20240501 14:00:48.944025 29101 catalog_manager.cc:5484] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:48.957175 29101 catalog_manager.cc:5484] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:48.969431 765 tablet_copy_client.cc:308] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:00:48.985020 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 578c3f4c5a7c4e2c962b64736195f0bc from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:50987)
I20240501 14:00:48.985214 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 578c3f4c5a7c4e2c962b64736195f0bc from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:50987: session id = 6edd6a3430d44c38a19c2804e555059f-578c3f4c5a7c4e2c962b64736195f0bc
I20240501 14:00:48.987715 29266 tablet_copy_source_session.cc:205] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.989141 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578c3f4c5a7c4e2c962b64736195f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.994410 765 tablet_copy_client.cc:783] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.994691 765 tablet_copy_client.cc:647] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.996706 765 tablet_copy_client.cc:523] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.999717 765 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:49.011322 29377 consensus_queue.cc:237] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.018540 29244 raft_consensus.cc:1261] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.020545 637 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:49.023874 29505 raft_consensus.cc:1261] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.024518 637 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:49.025492 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67ed6fab7d9e46aa8faeb3cf784af0fb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:49.031095 29101 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.031435 29101 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.036170 29101 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 8a7ab679e7324e70a1a9ac18fe38ad05
I20240501 14:00:49.037380 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:49.038147 788 tablet_replica.cc:330] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:49.038478 788 raft_consensus.cc:2229] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.038642 32340 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:49.039069 788 raft_consensus.cc:2258] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.040244 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7ab679e7324e70a1a9ac18fe38ad05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:49.045675 765 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.046170 765 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:00:49.046461 765 ts_tablet_manager.cc:1367] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.047s user 0.035s sys 0.011s
I20240501 14:00:49.047386 765 raft_consensus.cc:348] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.047609 765 raft_consensus.cc:729] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.048012 765 consensus_queue.cc:260] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.048576 765 ts_tablet_manager.cc:1398] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:49.049443 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-578c3f4c5a7c4e2c962b64736195f0bc received from {username='slave'} at 127.22.33.72:50987
I20240501 14:00:49.049603 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-578c3f4c5a7c4e2c962b64736195f0bc on tablet 578c3f4c5a7c4e2c962b64736195f0bc with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:49.058452 791 raft_consensus.cc:482] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.058640 791 raft_consensus.cc:504] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.059921 791 leader_election.cc:288] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.060037 29505 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:49.061074 789 tablet_replica.cc:330] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:49.061298 789 raft_consensus.cc:2229] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.061607 790 tablet_replica.cc:330] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:49.061879 790 raft_consensus.cc:2229] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:49.062397 790 raft_consensus.cc:2258] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.063231 788 ts_tablet_manager.cc:1875] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.065356 789 raft_consensus.cc:2258] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.069645 29101 catalog_manager.cc:5484] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:49.070397 789 ts_tablet_manager.cc:1875] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.070748 790 ts_tablet_manager.cc:1875] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.058916 792 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
W20240501 14:00:49.081763 29310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 56656.3us
I20240501 14:00:49.082985 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41986f1ae7648a5bfe7760c069af672" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:49.083264 29244 raft_consensus.cc:2454] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:49.083889 29310 leader_election.cc:302] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:49.084827 637 raft_consensus.cc:2781] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.084939 637 raft_consensus.cc:482] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.085101 637 raft_consensus.cc:3037] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.085304 790 ts_tablet_manager.cc:1888] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.085481 790 log.cc:1196] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/8a7ab679e7324e70a1a9ac18fe38ad05
I20240501 14:00:49.085954 790 ts_tablet_manager.cc:1909] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:49.087781 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table_new_name [id=914d070f147e41298d56fcfaf213b5c1]) successfully deleted
I20240501 14:00:49.088279 637 raft_consensus.cc:504] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.088546 789 ts_tablet_manager.cc:1888] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.088671 789 log.cc:1196] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/8a7ab679e7324e70a1a9ac18fe38ad05
I20240501 14:00:49.089005 637 leader_election.cc:288] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.089123 789 ts_tablet_manager.cc:1909] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:49.089442 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41986f1ae7648a5bfe7760c069af672" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:49.089663 29244 raft_consensus.cc:3037] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.090287 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table_new_name [id=914d070f147e41298d56fcfaf213b5c1]) successfully deleted
I20240501 14:00:49.092046 29244 raft_consensus.cc:2454] T e41986f1ae7648a5bfe7760c069af672 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:49.092504 29310 leader_election.cc:302] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:49.092842 637 raft_consensus.cc:2781] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.093324 637 raft_consensus.cc:686] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:49.093621 637 consensus_queue.cc:237] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.094906 788 ts_tablet_manager.cc:1888] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.095079 788 log.cc:1196] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/8a7ab679e7324e70a1a9ac18fe38ad05
I20240501 14:00:49.095541 788 ts_tablet_manager.cc:1909] T 8a7ab679e7324e70a1a9ac18fe38ad05 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:49.096778 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 8a7ab679e7324e70a1a9ac18fe38ad05 (table rebalancer_dynamic_table_new_name [id=914d070f147e41298d56fcfaf213b5c1]) successfully deleted
I20240501 14:00:49.097302 29101 catalog_manager.cc:5484] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:49.105450 29244 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:49.112257 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41986f1ae7648a5bfe7760c069af672" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:49.112499 31964 raft_consensus.cc:2454] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:49.113114 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41986f1ae7648a5bfe7760c069af672" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:49.113307 31965 raft_consensus.cc:3037] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.117617 31965 raft_consensus.cc:2454] T e41986f1ae7648a5bfe7760c069af672 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:49.149284 29101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:49.159281 29616 tablet_service.cc:1467] Processing CreateTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]), partition=
I20240501 14:00:49.159547 32340 tablet_service.cc:1467] Processing CreateTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]), partition=
I20240501 14:00:49.159722 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5792fbd44a84c83a205bace3b2ba2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.159915 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5792fbd44a84c83a205bace3b2ba2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.164273 732 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:49.164286 734 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:49.164566 29356 tablet_service.cc:1467] Processing CreateTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]), partition=
I20240501 14:00:49.164939 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5792fbd44a84c83a205bace3b2ba2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.165717 734 tablet_bootstrap.cc:654] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.165851 732 tablet_bootstrap.cc:654] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.172767 730 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:49.174942 730 tablet_bootstrap.cc:654] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.177371 732 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:49.177565 732 ts_tablet_manager.cc:1367] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240501 14:00:49.178226 732 raft_consensus.cc:348] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.178493 732 raft_consensus.cc:374] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.178575 732 raft_consensus.cc:729] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.178812 732 consensus_queue.cc:260] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.179319 732 ts_tablet_manager.cc:1398] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:49.184053 734 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:49.184365 734 ts_tablet_manager.cc:1367] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20240501 14:00:49.186170 734 raft_consensus.cc:348] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.186453 734 raft_consensus.cc:374] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.186599 734 raft_consensus.cc:729] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.186635 730 tablet_bootstrap.cc:492] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:49.187008 730 ts_tablet_manager.cc:1367] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240501 14:00:49.186909 734 consensus_queue.cc:260] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.187743 730 raft_consensus.cc:348] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.187952 730 raft_consensus.cc:374] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.188086 730 raft_consensus.cc:729] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.188361 730 consensus_queue.cc:260] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.188859 730 ts_tablet_manager.cc:1398] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:49.192395 734 ts_tablet_manager.cc:1398] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:49.197754 29505 consensus_queue.cc:237] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:00:49.200506 29442 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5893a54b753f4d52a900d34140f6363d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:49.201009 29244 raft_consensus.cc:1261] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.201815 29635 raft_consensus.cc:1261] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.202473 721 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:49.203388 721 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:49.206224 815 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:49.206873 29244 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:49.211956 29635 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:49.213135 29105 catalog_manager.cc:5484] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:49.218520 637 raft_consensus.cc:482] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.218690 637 raft_consensus.cc:504] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.219547 637 leader_election.cc:288] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:49.220703 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5792fbd44a84c83a205bace3b2ba2cd" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:49.220953 32362 raft_consensus.cc:2454] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:49.221357 29311 leader_election.cc:302] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:49.221423 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5792fbd44a84c83a205bace3b2ba2cd" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:49.221626 637 raft_consensus.cc:2781] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.221758 637 raft_consensus.cc:482] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.221863 637 raft_consensus.cc:3037] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.221984 29635 raft_consensus.cc:2454] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:49.223804 637 raft_consensus.cc:504] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.224378 637 leader_election.cc:288] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:49.224818 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5792fbd44a84c83a205bace3b2ba2cd" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:49.224835 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5792fbd44a84c83a205bace3b2ba2cd" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:49.225028 32362 raft_consensus.cc:3037] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.225065 29635 raft_consensus.cc:3037] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.227703 29635 raft_consensus.cc:2454] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:49.227710 32362 raft_consensus.cc:2454] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:49.228155 29311 leader_election.cc:302] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:49.228535 637 raft_consensus.cc:2781] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.228691 637 raft_consensus.cc:686] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:49.229192 637 consensus_queue.cc:237] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.233292 29105 catalog_manager.cc:5484] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:49.240332 670 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.246160 825 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.282516 29377 consensus_queue.cc:237] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:49.285178 29244 raft_consensus.cc:1261] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.286168 29505 raft_consensus.cc:1261] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:49.286646 792 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:49.286921 637 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:49.291376 791 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:49.293713 29505 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:00:49.295004 29311 consensus_peers.cc:485] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6e94a79c4514fe4a3ae8e41c7792dc9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:49.296227 29245 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:49.300254 29105 catalog_manager.cc:5484] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:49.338624 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:49.345285 849 ts_tablet_manager.cc:902] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.347050 32362 raft_consensus.cc:1261] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:49.347437 29635 raft_consensus.cc:1261] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:49.347829 792 consensus_queue.cc:1035] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.348068 791 consensus_queue.cc:1035] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.367458 29103 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]). Expected version 1 got 0
I20240501 14:00:49.370640 717 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:49.371311 717 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:49.369405 861 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:49.375555 589 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.375795 861 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:00:49.379396 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc] alter complete (version 1)
I20240501 14:00:49.421290 29103 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]). Expected version 1 got 0
I20240501 14:00:49.423141 750 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.458149 866 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:49.459293 866 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:49.460939 29103 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc]). Expected version 1 got 0
I20240501 14:00:49.469727 849 tablet_copy_client.cc:308] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:00:49.477667 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet 6c86d880c077411a99e6b9597adeeb0d from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:57827)
I20240501 14:00:49.477839 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet 6c86d880c077411a99e6b9597adeeb0d from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:57827: session id = b87d188552494b4590afc37a85b98c35-6c86d880c077411a99e6b9597adeeb0d
I20240501 14:00:49.480762 31985 tablet_copy_source_session.cc:205] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:49.485411 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:49.487757 849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c86d880c077411a99e6b9597adeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.498790 32362 raft_consensus.cc:1203] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:49.499284 849 tablet_copy_client.cc:783] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:49.499611 849 tablet_copy_client.cc:647] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:49.502933 861 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.503434 861 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:00:49.506384 866 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.506917 866 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:49.506966 717 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.507422 717 tablet.cc:1620] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:49.509085 849 tablet_copy_client.cc:523] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:49.516054 849 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:49.519901 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=0c83656dad3542fab20e5318a759b9bc] alter complete (version 2)
I20240501 14:00:49.529088 792 consensus_queue.cc:1035] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.550230 849 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.550783 849 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:00:49.551054 849 ts_tablet_manager.cc:1367] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.007s
I20240501 14:00:49.551888 792 consensus_queue.cc:1035] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.552263 849 raft_consensus.cc:348] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.552488 849 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.552819 849 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:49.553535 849 ts_tablet_manager.cc:1398] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:49.555266 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-6c86d880c077411a99e6b9597adeeb0d received from {username='slave'} at 127.22.33.66:57827
I20240501 14:00:49.555410 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session b87d188552494b4590afc37a85b98c35-6c86d880c077411a99e6b9597adeeb0d on tablet 6c86d880c077411a99e6b9597adeeb0d with peer b87d188552494b4590afc37a85b98c35
I20240501 14:00:49.617239 886 ts_tablet_manager.cc:902] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:49.624811 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:49.635134 886 tablet_copy_client.cc:308] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:00:49.636212 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:49865)
I20240501 14:00:49.636401 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 67ed6fab7d9e46aa8faeb3cf784af0fb from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:49865: session id = 6edd6a3430d44c38a19c2804e555059f-67ed6fab7d9e46aa8faeb3cf784af0fb
I20240501 14:00:49.638628 866 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.639012 717 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.639139 29397 tablet_copy_source_session.cc:205] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:49.640120 861 tablet.cc:1722] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.648072 886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.651626 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=0c83656dad3542fab20e5318a759b9bc] alter complete (version 3)
I20240501 14:00:49.656118 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:49.658790 886 tablet_copy_client.cc:783] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:49.659785 886 tablet_copy_client.cc:647] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:49.664762 886 tablet_copy_client.cc:523] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:49.668349 886 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:49.670125 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6bff08fa14cf6b4894f58e248fb59 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.670517 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da6bff08fa14cf6b4894f58e248fb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.671125 29224 tablet_service.cc:1467] Processing CreateTablet for tablet c696bdff78504c0c93f39a7e07a40dcd (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.671521 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c696bdff78504c0c93f39a7e07a40dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.673821 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 46d49324c56d499a96d7cca9af8a4867 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.674234 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d49324c56d499a96d7cca9af8a4867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.677958 31945 tablet_service.cc:1467] Processing CreateTablet for tablet c696bdff78504c0c93f39a7e07a40dcd (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.678179 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6bff08fa14cf6b4894f58e248fb59 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.678437 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c696bdff78504c0c93f39a7e07a40dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.678617 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da6bff08fa14cf6b4894f58e248fb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.681352 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 46d49324c56d499a96d7cca9af8a4867 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.681758 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d49324c56d499a96d7cca9af8a4867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.685550 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6bff08fa14cf6b4894f58e248fb59 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.686004 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da6bff08fa14cf6b4894f58e248fb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.689553 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 46d49324c56d499a96d7cca9af8a4867 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.690070 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d49324c56d499a96d7cca9af8a4867. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.690299 894 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:49.691965 894 tablet_bootstrap.cc:654] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.694084 894 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:49.694224 894 ts_tablet_manager.cc:1367] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:49.694191 32208 tablet_service.cc:1467] Processing CreateTablet for tablet c696bdff78504c0c93f39a7e07a40dcd (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.694718 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c696bdff78504c0c93f39a7e07a40dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.695083 894 raft_consensus.cc:348] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.695291 894 raft_consensus.cc:374] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.695394 894 raft_consensus.cc:729] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.695654 894 consensus_queue.cc:260] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.696251 894 ts_tablet_manager.cc:1398] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:49.696594 894 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:49.698012 896 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:49.698849 894 tablet_bootstrap.cc:654] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.699682 896 tablet_bootstrap.cc:654] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.708891 900 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:49.712303 900 tablet_bootstrap.cc:654] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.724202 900 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:49.724762 900 ts_tablet_manager.cc:1367] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240501 14:00:49.725919 900 raft_consensus.cc:348] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:49.727443 900 raft_consensus.cc:374] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.727739 900 raft_consensus.cc:729] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.728284 900 consensus_queue.cc:260] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:49.729396 900 ts_tablet_manager.cc:1398] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:49.753742 902 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:49.755597 902 tablet_bootstrap.cc:654] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.763726 886 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.764417 886 tablet_bootstrap.cc:492] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:00:49.765251 886 ts_tablet_manager.cc:1367] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.097s user 0.068s sys 0.004s
I20240501 14:00:49.766484 902 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:49.766151 29101 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.766551 886 raft_consensus.cc:348] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.766875 886 raft_consensus.cc:729] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.766743 29101 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.767292 886 consensus_queue.cc:260] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.767907 902 ts_tablet_manager.cc:1367] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 14:00:49.768285 886 ts_tablet_manager.cc:1398] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:49.769248 902 raft_consensus.cc:348] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.769425 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-67ed6fab7d9e46aa8faeb3cf784af0fb received from {username='slave'} at 127.22.33.72:49865
I20240501 14:00:49.770362 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-67ed6fab7d9e46aa8faeb3cf784af0fb on tablet 67ed6fab7d9e46aa8faeb3cf784af0fb with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:49.771030 902 raft_consensus.cc:374] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.771298 902 raft_consensus.cc:729] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.771814 902 consensus_queue.cc:260] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.773382 902 ts_tablet_manager.cc:1398] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:49.795140 29101 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet e5792fbd44a84c83a205bace3b2ba2cd
I20240501 14:00:49.796952 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:49.800208 29377 raft_consensus.cc:1203] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:49.803659 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:49.804286 909 tablet_replica.cc:330] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:49.804502 909 raft_consensus.cc:2229] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.804800 909 raft_consensus.cc:2258] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.805536 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet e5792fbd44a84c83a205bace3b2ba2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:49.806265 909 ts_tablet_manager.cc:1875] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.806720 910 tablet_replica.cc:330] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:49.807049 910 raft_consensus.cc:2229] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:49.807529 910 raft_consensus.cc:2258] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.809156 910 ts_tablet_manager.cc:1875] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.811753 903 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:49.813838 903 tablet_bootstrap.cc:654] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.815174 907 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:49.816917 907 tablet_bootstrap.cc:654] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.817464 904 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:49.819272 904 tablet_bootstrap.cc:654] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.826128 906 ts_tablet_manager.cc:902] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:49.829191 908 tablet_replica.cc:330] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:49.829612 908 raft_consensus.cc:2229] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.830192 908 raft_consensus.cc:2258] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.831526 903 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:49.831790 903 ts_tablet_manager.cc:1367] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240501 14:00:49.832897 903 raft_consensus.cc:348] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.833294 903 raft_consensus.cc:374] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.833472 903 raft_consensus.cc:729] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.833829 903 consensus_queue.cc:260] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.834558 903 ts_tablet_manager.cc:1398] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:49.851784 907 tablet_bootstrap.cc:492] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:49.852092 907 ts_tablet_manager.cc:1367] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.006s
I20240501 14:00:49.864418 909 ts_tablet_manager.cc:1888] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.864682 909 log.cc:1196] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5792fbd44a84c83a205bace3b2ba2cd
I20240501 14:00:49.865227 908 ts_tablet_manager.cc:1875] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.865356 909 ts_tablet_manager.cc:1909] T e5792fbd44a84c83a205bace3b2ba2cd P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:49.866686 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table_new_name [id=0c83656dad3542fab20e5318a759b9bc]) successfully deleted
I20240501 14:00:49.872265 910 ts_tablet_manager.cc:1888] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.881125 910 log.cc:1196] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5792fbd44a84c83a205bace3b2ba2cd
I20240501 14:00:49.882028 910 ts_tablet_manager.cc:1909] T e5792fbd44a84c83a205bace3b2ba2cd P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:49.882941 896 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:49.883184 896 ts_tablet_manager.cc:1367] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.185s user 0.025s sys 0.063s
I20240501 14:00:49.884481 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table_new_name [id=0c83656dad3542fab20e5318a759b9bc]) successfully deleted
I20240501 14:00:49.885066 896 raft_consensus.cc:348] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.886350 896 raft_consensus.cc:374] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.886528 896 raft_consensus.cc:729] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.886894 896 consensus_queue.cc:260] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.888619 896 ts_tablet_manager.cc:1398] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:49.890699 907 raft_consensus.cc:348] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.891125 907 raft_consensus.cc:374] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.891304 907 raft_consensus.cc:729] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.891706 907 consensus_queue.cc:260] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:49.894862 907 ts_tablet_manager.cc:1398] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.026s user 0.000s sys 0.003s
I20240501 14:00:49.895258 907 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:49.898679 894 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:49.898898 894 ts_tablet_manager.cc:1367] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.202s user 0.047s sys 0.056s
I20240501 14:00:49.899751 894 raft_consensus.cc:348] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.899989 894 raft_consensus.cc:374] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.900076 894 raft_consensus.cc:729] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.900326 894 consensus_queue.cc:260] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.902343 894 ts_tablet_manager.cc:1398] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:49.907171 904 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:49.906740 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:49.907610 904 ts_tablet_manager.cc:1367] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.000s
I20240501 14:00:49.908550 904 raft_consensus.cc:348] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:49.908860 904 raft_consensus.cc:374] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.909034 904 raft_consensus.cc:729] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.909385 904 consensus_queue.cc:260] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:49.911152 904 ts_tablet_manager.cc:1398] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:49.913712 750 raft_consensus.cc:482] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.913959 750 raft_consensus.cc:504] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.914649 750 leader_election.cc:288] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.916425 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:49.916702 31965 raft_consensus.cc:2454] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:49.917130 32163 leader_election.cc:302] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:49.917642 750 raft_consensus.cc:2781] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.917745 750 raft_consensus.cc:482] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.917847 750 raft_consensus.cc:3037] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.920521 750 raft_consensus.cc:504] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.922713 750 leader_election.cc:288] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.923383 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:49.923624 31965 raft_consensus.cc:3037] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.926546 31965 raft_consensus.cc:2454] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:49.927067 32163 leader_election.cc:302] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:49.927448 750 raft_consensus.cc:2781] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.932426 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 004801682b7a48e0b185d2511086f1af (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]), partition=
I20240501 14:00:49.932914 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004801682b7a48e0b185d2511086f1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.938937 906 tablet_copy_client.cc:308] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:00:49.940246 32077 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:49.941079 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 004801682b7a48e0b185d2511086f1af (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]), partition=
I20240501 14:00:49.941579 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004801682b7a48e0b185d2511086f1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.950007 941 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:49.951304 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 004801682b7a48e0b185d2511086f1af (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]), partition=
I20240501 14:00:49.951730 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004801682b7a48e0b185d2511086f1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.952991 941 tablet_bootstrap.cc:654] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.954344 651 raft_consensus.cc:482] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.954494 651 raft_consensus.cc:504] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:49.955267 651 leader_election.cc:288] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:49.955910 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:49.958122 904 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:49.959681 904 tablet_bootstrap.cc:654] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.972056 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:49.972328 29245 raft_consensus.cc:3037] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.973201 371 raft_consensus.cc:482] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.973425 371 raft_consensus.cc:504] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:49.974553 371 leader_election.cc:288] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:49.975800 29245 raft_consensus.cc:2454] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:49.976270 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:49.976531 29244 raft_consensus.cc:2362] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1. Re-sending same reply.
I20240501 14:00:49.976594 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d49324c56d499a96d7cca9af8a4867" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:00:49.977600 32296 leader_election.cc:341] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:49.953809 896 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:49.957522 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c696bdff78504c0c93f39a7e07a40dcd" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:49.986310 896 tablet_bootstrap.cc:654] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.987179 904 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:49.987319 31965 raft_consensus.cc:2379] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6518b7a04854458bb2fb23edc8b6f7ba in current term 1: Already voted for candidate 6a446ac802a7457d9f27076781f21d79 in this term.
I20240501 14:00:49.987401 904 ts_tablet_manager.cc:1367] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20240501 14:00:49.988715 896 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:49.988593 904 raft_consensus.cc:348] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.988893 896 ts_tablet_manager.cc:1367] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240501 14:00:49.989073 904 raft_consensus.cc:374] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.989228 904 raft_consensus.cc:729] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.989543 750 raft_consensus.cc:686] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:49.990065 896 raft_consensus.cc:348] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.990298 896 raft_consensus.cc:374] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.990382 896 raft_consensus.cc:729] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.990659 896 consensus_queue.cc:260] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:49.991235 934 ts_tablet_manager.cc:902] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:49.992192 896 ts_tablet_manager.cc:1398] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:49.999907 32341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:50.004357 750 consensus_queue.cc:237] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:50.010219 29226 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:50.010378 29180 leader_election.cc:302] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba; no voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:50.010833 651 raft_consensus.cc:2726] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:50.010991 637 raft_consensus.cc:482] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.011179 637 raft_consensus.cc:504] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:50.012038 637 leader_election.cc:288] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:50.012603 757 raft_consensus.cc:482] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.012679 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:50.012765 757 raft_consensus.cc:504] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.012917 29245 raft_consensus.cc:2454] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:50.013391 29310 leader_election.cc:302] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:50.013597 757 leader_election.cc:288] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:50.013904 637 raft_consensus.cc:2781] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.014017 637 raft_consensus.cc:482] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.014098 637 raft_consensus.cc:3037] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.017786 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004801682b7a48e0b185d2511086f1af" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
W20240501 14:00:50.018383 31898 leader_election.cc:341] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.022084 637 raft_consensus.cc:504] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:50.024113 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:50.024334 29244 raft_consensus.cc:3037] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.027607 32208 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:50.028152 29244 raft_consensus.cc:2454] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:50.028373 32360 raft_consensus.cc:1203] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:50.028610 29310 leader_election.cc:302] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:50.028940 791 raft_consensus.cc:2781] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.029138 791 raft_consensus.cc:686] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:50.029428 791 consensus_queue.cc:237] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:50.029556 904 consensus_queue.cc:260] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.029915 29101 catalog_manager.cc:5484] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:50.031553 904 ts_tablet_manager.cc:1398] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240501 14:00:50.031801 637 leader_election.cc:288] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:50.055155 29101 catalog_manager.cc:5484] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:50.069568 907 tablet_bootstrap.cc:654] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.082089 31945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:50.083926 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004801682b7a48e0b185d2511086f1af" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:50.084201 32360 raft_consensus.cc:2454] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:50.084683 31899 leader_election.cc:302] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:50.085201 757 raft_consensus.cc:2781] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.085326 757 raft_consensus.cc:482] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.085453 757 raft_consensus.cc:3037] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.087903 757 raft_consensus.cc:504] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.088881 757 leader_election.cc:288] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:50.089454 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004801682b7a48e0b185d2511086f1af" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:50.089550 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004801682b7a48e0b185d2511086f1af" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:50.089782 32360 raft_consensus.cc:3037] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:50.090158 31898 leader_election.cc:341] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.097379 32360 raft_consensus.cc:2454] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:50.109484 31899 leader_election.cc:302] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:50.109877 757 raft_consensus.cc:2781] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.122324 757 raft_consensus.cc:686] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:50.122848 757 consensus_queue.cc:237] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.127187 908 ts_tablet_manager.cc:1888] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.127475 908 log.cc:1196] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/e5792fbd44a84c83a205bace3b2ba2cd
I20240501 14:00:50.128635 908 ts_tablet_manager.cc:1909] T e5792fbd44a84c83a205bace3b2ba2cd P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:50.132578 907 tablet_bootstrap.cc:492] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:50.132957 907 ts_tablet_manager.cc:1367] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.238s user 0.005s sys 0.000s
I20240501 14:00:50.133877 825 raft_consensus.cc:482] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.134110 825 raft_consensus.cc:504] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:50.135177 825 leader_election.cc:288] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:50.135390 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 5893a54b753f4d52a900d34140f6363d from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:54045)
I20240501 14:00:50.135600 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 5893a54b753f4d52a900d34140f6363d from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:54045: session id = 9903ed35bca144bcb74e9bf5102117e7-5893a54b753f4d52a900d34140f6363d
I20240501 14:00:50.138700 29527 tablet_copy_source_session.cc:205] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.139053 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:50.139485 29244 raft_consensus.cc:2379] T 7da6bff08fa14cf6b4894f58e248fb59 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a697f3a1ee544daa7073814ab845d58 in current term 1: Already voted for candidate b87d188552494b4590afc37a85b98c35 in this term.
I20240501 14:00:50.146514 906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5893a54b753f4d52a900d34140f6363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.149056 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:50.156368 941 tablet_bootstrap.cc:492] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:50.159451 29101 catalog_manager.cc:5484] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:50.162644 941 ts_tablet_manager.cc:1367] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.213s user 0.004s sys 0.000s
I20240501 14:00:50.169435 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d49324c56d499a96d7cca9af8a4867" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:50.169715 29356 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:50.169728 29505 raft_consensus.cc:2454] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:50.171043 934 tablet_copy_client.cc:308] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:00:50.180219 29616 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:50.210414 941 raft_consensus.cc:348] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.213452 907 raft_consensus.cc:348] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:50.221879 941 raft_consensus.cc:374] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.222461 941 raft_consensus.cc:729] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.224001 907 raft_consensus.cc:374] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.224180 907 raft_consensus.cc:729] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.225332 941 consensus_queue.cc:260] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:50.227645 941 ts_tablet_manager.cc:1398] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20240501 14:00:50.170233 32295 leader_election.cc:302] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:50.228417 371 raft_consensus.cc:2781] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.228578 371 raft_consensus.cc:482] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.228682 371 raft_consensus.cc:3037] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.231575 371 raft_consensus.cc:504] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:50.232688 371 leader_election.cc:288] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
W20240501 14:00:50.234360 32295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 64162.7us
I20240501 14:00:50.235749 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet a6e94a79c4514fe4a3ae8e41c7792dc9 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:57801)
I20240501 14:00:50.235749 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d49324c56d499a96d7cca9af8a4867" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:50.235903 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet a6e94a79c4514fe4a3ae8e41c7792dc9 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:57801: session id = 6a446ac802a7457d9f27076781f21d79-a6e94a79c4514fe4a3ae8e41c7792dc9
I20240501 14:00:50.236385 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d49324c56d499a96d7cca9af8a4867" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:50.236598 29504 raft_consensus.cc:3037] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.238797 29397 tablet_copy_source_session.cc:205] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.239646 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet e5792fbd44a84c83a205bace3b2ba2cd (table rebalancer_dynamic_table_new_name [id=0c83656dad3542fab20e5318a759b9bc]) successfully deleted
I20240501 14:00:50.239742 29504 raft_consensus.cc:2454] T 46d49324c56d499a96d7cca9af8a4867 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:50.240226 32295 leader_election.cc:302] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:50.240650 371 raft_consensus.cc:2781] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.241022 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:50.241885 32032 leader_election.cc:302] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:00:50.242445 963 raft_consensus.cc:3037] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.244637 963 raft_consensus.cc:2726] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:50.245290 934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e94a79c4514fe4a3ae8e41c7792dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.253211 371 raft_consensus.cc:686] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:50.255060 371 consensus_queue.cc:237] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:50.224562 907 consensus_queue.cc:260] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:50.267091 32097 raft_consensus.cc:3037] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.270332 32097 raft_consensus.cc:2454] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:50.272753 907 ts_tablet_manager.cc:1398] T 46d49324c56d499a96d7cca9af8a4867 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.140s user 0.003s sys 0.000s
I20240501 14:00:50.280613 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:50.281061 32097 raft_consensus.cc:2454] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:50.283409 29101 catalog_manager.cc:5484] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:50.284169 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da6bff08fa14cf6b4894f58e248fb59" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:50.286834 934 tablet_copy_client.cc:783] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.287302 934 tablet_copy_client.cc:647] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.289315 32097 raft_consensus.cc:2454] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:50.289579 934 tablet_copy_client.cc:523] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:50.293308 906 tablet_copy_client.cc:783] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.293562 906 tablet_copy_client.cc:647] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
W20240501 14:00:50.294633 31899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 145594us
I20240501 14:00:50.299041 906 tablet_copy_client.cc:523] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:50.300194 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:50.311031 906 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:50.324630 934 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:50.344803 32362 raft_consensus.cc:1261] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.345640 956 consensus_queue.cc:1035] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.352149 29101 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]). Expected version 1 got 0
I20240501 14:00:50.369041 29635 raft_consensus.cc:3037] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.372965 29635 raft_consensus.cc:1261] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.374442 956 consensus_queue.cc:1035] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.384049 972 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.387207 29101 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]). Expected version 1 got 0
I20240501 14:00:50.395012 971 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.395644 971 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:50.397514 972 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:50.400146 906 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:50.400728 906 tablet_bootstrap.cc:492] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:00:50.401177 906 ts_tablet_manager.cc:1367] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.090s user 0.046s sys 0.020s
I20240501 14:00:50.402374 906 raft_consensus.cc:348] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:50.402761 906 raft_consensus.cc:729] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.403426 976 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.404089 976 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:50.404561 906 consensus_queue.cc:260] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:50.405213 906 ts_tablet_manager.cc:1398] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:50.406301 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-5893a54b753f4d52a900d34140f6363d received from {username='slave'} at 127.22.33.69:54045
I20240501 14:00:50.406445 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-5893a54b753f4d52a900d34140f6363d on tablet 5893a54b753f4d52a900d34140f6363d with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:50.412777 29101 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313]). Expected version 1 got 0
I20240501 14:00:50.420212 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313] alter complete (version 1)
I20240501 14:00:50.423746 949 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.469542 934 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:50.469599 949 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.470098 934 tablet_bootstrap.cc:492] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:00:50.470422 934 ts_tablet_manager.cc:1367] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.159s user 0.066s sys 0.017s
I20240501 14:00:50.472380 934 raft_consensus.cc:348] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:50.472723 934 raft_consensus.cc:729] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.473035 934 consensus_queue.cc:260] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:50.478077 934 ts_tablet_manager.cc:1398] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:00:50.482425 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-a6e94a79c4514fe4a3ae8e41c7792dc9 received from {username='slave'} at 127.22.33.71:57801
I20240501 14:00:50.482589 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-a6e94a79c4514fe4a3ae8e41c7792dc9 on tablet a6e94a79c4514fe4a3ae8e41c7792dc9 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:50.518743 637 consensus_queue.cc:1035] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.529191 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:50.551051 971 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.551606 971 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:50.559563 976 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.560102 976 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:50.563506 972 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.564018 972 tablet.cc:1620] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:50.568001 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f687e1806c8b4937b13f2aea35c8d313] alter complete (version 2)
I20240501 14:00:50.571494 637 consensus_queue.cc:1035] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.590972 31965 raft_consensus.cc:1203] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:50.639819 29635 consensus_queue.cc:237] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:50.642824 32362 raft_consensus.cc:1261] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:50.643179 29572 consensus_peers.cc:485] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5e2a1f3418c477aa65b77bfa171a5a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.643658 501 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.646107 32097 raft_consensus.cc:1261] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.646643 501 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.649132 1003 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:50.650068 32097 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:50.654444 32362 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:50.655727 29096 catalog_manager.cc:5484] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:50.675705 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:50.684099 971 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.684706 972 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.685559 976 tablet.cc:1722] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.695430 29102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=f687e1806c8b4937b13f2aea35c8d313] alter complete (version 3)
I20240501 14:00:50.704703 32097 consensus_queue.cc:237] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
W20240501 14:00:50.707504 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee3fe52d2ae462a823d3ac03f2ce8bc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.707659 32228 raft_consensus.cc:1261] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.707770 29635 raft_consensus.cc:1261] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.708367 1022 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.708961 963 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.711409 963 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:50.711946 29635 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:50.716136 32228 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:50.717114 29105 catalog_manager.cc:5484] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:50.755663 29504 consensus_queue.cc:237] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:50.758268 29244 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:50.758534 29377 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:50.758884 577 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:50.759249 29441 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ffd2100f104b339bfdc2d272ba3729. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.759456 1042 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:50.761889 577 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:50.762482 29377 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:50.768743 29244 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:50.770164 29102 catalog_manager.cc:5484] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:50.801808 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:50.802057 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:50.805826 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 004801682b7a48e0b185d2511086f1af
I20240501 14:00:50.806070 32228 consensus_queue.cc:237] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:50.806854 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet 004801682b7a48e0b185d2511086f1af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:50.807003 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 004801682b7a48e0b185d2511086f1af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:50.807317 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 004801682b7a48e0b185d2511086f1af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:50.808874 32227 raft_consensus.cc:1203] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:50.809003 29244 raft_consensus.cc:1261] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:50.809453 32163 consensus_peers.cc:485] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 820be6be18af4781ace626f239379cff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.809723 750 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.812562 1064 tablet_replica.cc:330] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:50.812932 1064 raft_consensus.cc:2229] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.813033 32097 raft_consensus.cc:1261] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.813314 1064 raft_consensus.cc:2258] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.814314 1064 ts_tablet_manager.cc:1875] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.814190 750 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:50.823513 1061 tablet_replica.cc:330] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:50.823913 1061 raft_consensus.cc:2229] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.824285 1061 raft_consensus.cc:2258] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.825575 1062 tablet_replica.cc:330] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:50.825584 1061 ts_tablet_manager.cc:1875] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.825809 1062 raft_consensus.cc:2229] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:50.826212 1062 raft_consensus.cc:2258] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.826696 371 consensus_queue.cc:1035] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.828464 1062 ts_tablet_manager.cc:1875] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.834918 1061 ts_tablet_manager.cc:1888] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.835141 1061 log.cc:1196] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/004801682b7a48e0b185d2511086f1af
I20240501 14:00:50.835681 1061 ts_tablet_manager.cc:1909] T 004801682b7a48e0b185d2511086f1af P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:50.835884 750 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:50.837112 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table_new_name [id=f687e1806c8b4937b13f2aea35c8d313]) successfully deleted
I20240501 14:00:50.839413 1062 ts_tablet_manager.cc:1888] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.839547 1062 log.cc:1196] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/004801682b7a48e0b185d2511086f1af
I20240501 14:00:50.839977 1062 ts_tablet_manager.cc:1909] T 004801682b7a48e0b185d2511086f1af P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:50.840487 32097 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:50.843350 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table_new_name [id=f687e1806c8b4937b13f2aea35c8d313]) successfully deleted
I20240501 14:00:50.846114 29105 catalog_manager.cc:5484] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:50.853188 29244 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:50.860589 371 consensus_queue.cc:1035] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.869315 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:50.872427 1064 ts_tablet_manager.cc:1888] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.872615 1064 log.cc:1196] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/004801682b7a48e0b185d2511086f1af
I20240501 14:00:50.883677 1064 ts_tablet_manager.cc:1909] T 004801682b7a48e0b185d2511086f1af P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:50.885550 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 004801682b7a48e0b185d2511086f1af (table rebalancer_dynamic_table_new_name [id=f687e1806c8b4937b13f2aea35c8d313]) successfully deleted
I20240501 14:00:50.888954 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.889591 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.890425 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.890879 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.890791 31945 tablet_service.cc:1467] Processing CreateTablet for tablet bfbf43ae81ae45ba8d5abd345d18ac8a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.891191 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbf43ae81ae45ba8d5abd345d18ac8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.893342 32207 tablet_service.cc:1467] Processing CreateTablet for tablet bfbf43ae81ae45ba8d5abd345d18ac8a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.893601 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 4d374fd761284e32913896ac5a22ab63 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.893745 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbf43ae81ae45ba8d5abd345d18ac8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.894043 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d374fd761284e32913896ac5a22ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.897338 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 4d374fd761284e32913896ac5a22ab63 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.897431 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.897677 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d374fd761284e32913896ac5a22ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.897840 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.902297 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 4d374fd761284e32913896ac5a22ab63 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.902426 32076 tablet_service.cc:1467] Processing CreateTablet for tablet bfbf43ae81ae45ba8d5abd345d18ac8a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.902745 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d374fd761284e32913896ac5a22ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.902895 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbf43ae81ae45ba8d5abd345d18ac8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.921969 1077 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:50.923758 1077 tablet_bootstrap.cc:654] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.929301 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:50.931852 1077 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:50.940598 1076 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:50.948428 1075 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:50.950914 1075 tablet_bootstrap.cc:654] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.954952 1081 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:50.956614 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 127424670b7549ecab23639640d4e605 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]), partition=
I20240501 14:00:50.956578 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 127424670b7549ecab23639640d4e605 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]), partition=
I20240501 14:00:50.956964 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127424670b7549ecab23639640d4e605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.957619 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 127424670b7549ecab23639640d4e605 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]), partition=
I20240501 14:00:50.958014 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127424670b7549ecab23639640d4e605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.959985 1081 tablet_bootstrap.cc:654] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.961167 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127424670b7549ecab23639640d4e605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.964640 1075 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:50.965623 1077 ts_tablet_manager.cc:1367] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240501 14:00:50.966321 1075 ts_tablet_manager.cc:1367] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240501 14:00:50.967494 1075 raft_consensus.cc:348] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:50.971463 1075 raft_consensus.cc:374] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.971900 1075 raft_consensus.cc:729] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.971958 1076 tablet_bootstrap.cc:654] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.966822 1077 raft_consensus.cc:348] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:50.973425 1077 raft_consensus.cc:374] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.973630 1077 raft_consensus.cc:729] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.973978 1077 consensus_queue.cc:260] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:50.974898 1077 ts_tablet_manager.cc:1398] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:50.988659 1073 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:50.990875 1073 tablet_bootstrap.cc:654] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.993312 1076 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:50.993638 1076 ts_tablet_manager.cc:1367] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.000s
I20240501 14:00:50.994738 1076 raft_consensus.cc:348] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:50.995175 1076 raft_consensus.cc:374] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.995333 1076 raft_consensus.cc:729] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.995656 1076 consensus_queue.cc:260] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:50.996362 1076 ts_tablet_manager.cc:1398] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:50.986763 1078 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:51.000798 1078 tablet_bootstrap.cc:654] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.001693 1081 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:51.002153 1075 consensus_queue.cc:260] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.004323 1075 ts_tablet_manager.cc:1398] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240501 14:00:51.004586 1081 ts_tablet_manager.cc:1367] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240501 14:00:51.006889 1081 raft_consensus.cc:348] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.007154 1081 raft_consensus.cc:374] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.007297 1081 raft_consensus.cc:729] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.007579 1081 consensus_queue.cc:260] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.008195 1081 ts_tablet_manager.cc:1398] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:51.008533 1081 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:51.006035 1075 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:51.012640 1081 tablet_bootstrap.cc:654] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.012830 1075 tablet_bootstrap.cc:654] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.032444 1073 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:51.032647 1073 ts_tablet_manager.cc:1367] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240501 14:00:51.033651 1081 tablet_bootstrap.cc:492] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:51.033838 1081 ts_tablet_manager.cc:1367] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.009s
I20240501 14:00:51.034647 1081 raft_consensus.cc:348] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.034888 1081 raft_consensus.cc:374] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.034970 1081 raft_consensus.cc:729] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.035212 1081 consensus_queue.cc:260] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.035485 1075 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:51.035686 1075 ts_tablet_manager.cc:1367] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.011s
I20240501 14:00:51.035842 1081 ts_tablet_manager.cc:1398] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:51.036149 1081 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:51.036592 1075 raft_consensus.cc:348] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.036830 1075 raft_consensus.cc:374] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.036706 1073 raft_consensus.cc:348] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.036928 1075 raft_consensus.cc:729] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.037024 1073 raft_consensus.cc:374] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.037145 1073 raft_consensus.cc:729] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.037211 1075 consensus_queue.cc:260] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.037398 1073 consensus_queue.cc:260] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.037722 1075 ts_tablet_manager.cc:1398] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:51.038110 1081 tablet_bootstrap.cc:654] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.038125 1073 ts_tablet_manager.cc:1398] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:51.038508 1073 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:51.041776 1073 tablet_bootstrap.cc:654] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.042709 1078 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:51.042945 1078 ts_tablet_manager.cc:1367] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240501 14:00:51.045343 1078 raft_consensus.cc:348] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.045719 1078 raft_consensus.cc:374] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.045872 1078 raft_consensus.cc:729] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.046238 1078 consensus_queue.cc:260] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.047737 1078 ts_tablet_manager.cc:1398] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:51.048173 1078 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:51.050078 1078 tablet_bootstrap.cc:654] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.046551 1081 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:51.050415 1081 ts_tablet_manager.cc:1367] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240501 14:00:51.054438 1081 raft_consensus.cc:348] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.054744 1081 raft_consensus.cc:374] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.054888 1081 raft_consensus.cc:729] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.055274 1081 consensus_queue.cc:260] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.060065 1081 ts_tablet_manager.cc:1398] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:00:51.070554 1073 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:51.070833 1073 ts_tablet_manager.cc:1367] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240501 14:00:51.073304 750 raft_consensus.cc:482] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.073493 750 raft_consensus.cc:504] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.073833 1073 raft_consensus.cc:348] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.074062 1073 raft_consensus.cc:374] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.074330 750 leader_election.cc:288] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:51.074136 1073 raft_consensus.cc:729] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.074936 1073 consensus_queue.cc:260] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.077140 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d374fd761284e32913896ac5a22ab63" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:51.077376 32362 raft_consensus.cc:2454] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:51.078034 32162 leader_election.cc:302] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:51.078081 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d374fd761284e32913896ac5a22ab63" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:51.078325 949 raft_consensus.cc:2781] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.078325 29504 raft_consensus.cc:2454] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:51.078421 949 raft_consensus.cc:482] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.078519 949 raft_consensus.cc:3037] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.081642 949 raft_consensus.cc:504] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.083135 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d374fd761284e32913896ac5a22ab63" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:51.083365 32362 raft_consensus.cc:3037] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.086326 32362 raft_consensus.cc:2454] T 4d374fd761284e32913896ac5a22ab63 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:51.086791 32162 leader_election.cc:302] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:51.087021 501 raft_consensus.cc:482] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.087304 750 raft_consensus.cc:2781] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.087215 501 raft_consensus.cc:504] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.087989 501 leader_election.cc:288] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.088678 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec9bb1d59ac4b11a1f9b0d44b1dc69b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:51.088897 32096 raft_consensus.cc:2454] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:51.089633 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec9bb1d59ac4b11a1f9b0d44b1dc69b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:51.089833 32228 raft_consensus.cc:2454] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:51.090374 29572 leader_election.cc:302] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:51.090679 1073 ts_tablet_manager.cc:1398] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:51.091156 501 raft_consensus.cc:2781] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.091259 501 raft_consensus.cc:482] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.091326 501 raft_consensus.cc:3037] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.094179 501 raft_consensus.cc:504] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.094538 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d374fd761284e32913896ac5a22ab63" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:51.094748 29504 raft_consensus.cc:3037] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.095314 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec9bb1d59ac4b11a1f9b0d44b1dc69b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:51.095515 32228 raft_consensus.cc:3037] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.097865 29504 raft_consensus.cc:2454] T 4d374fd761284e32913896ac5a22ab63 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:51.098284 32228 raft_consensus.cc:2454] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:51.100471 949 leader_election.cc:288] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:51.100682 29571 leader_election.cc:302] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:51.101025 1003 raft_consensus.cc:2781] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.101200 1003 raft_consensus.cc:686] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:51.101446 1003 consensus_queue.cc:237] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:51.103855 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec9bb1d59ac4b11a1f9b0d44b1dc69b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:51.104092 32096 raft_consensus.cc:3037] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.107023 32096 raft_consensus.cc:2454] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:51.108784 501 leader_election.cc:288] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.121253 750 raft_consensus.cc:686] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:51.121728 750 consensus_queue.cc:237] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.127852 29096 catalog_manager.cc:5484] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:51.129065 1078 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:51.129269 1078 ts_tablet_manager.cc:1367] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.018s
I20240501 14:00:51.137373 1078 raft_consensus.cc:348] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.138175 1078 raft_consensus.cc:374] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.142092 1078 raft_consensus.cc:729] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.142491 1078 consensus_queue.cc:260] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.143970 1078 ts_tablet_manager.cc:1398] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:00:51.143821 29096 catalog_manager.cc:5484] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:51.148097 1078 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:51.149989 1078 tablet_bootstrap.cc:654] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.153844 757 raft_consensus.cc:482] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.154028 757 raft_consensus.cc:504] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.154726 757 leader_election.cc:288] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.155200 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbf43ae81ae45ba8d5abd345d18ac8a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:51.155428 32096 raft_consensus.cc:2454] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:51.155771 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbf43ae81ae45ba8d5abd345d18ac8a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:51.155987 32228 raft_consensus.cc:2454] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:51.156327 31900 leader_election.cc:302] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:51.156638 757 raft_consensus.cc:2781] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.156737 757 raft_consensus.cc:482] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.156836 757 raft_consensus.cc:3037] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.159883 757 raft_consensus.cc:504] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.160562 757 leader_election.cc:288] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.161084 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbf43ae81ae45ba8d5abd345d18ac8a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:51.161279 32096 raft_consensus.cc:3037] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.161363 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbf43ae81ae45ba8d5abd345d18ac8a" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:51.161597 32227 raft_consensus.cc:3037] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.164075 32096 raft_consensus.cc:2454] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:51.164397 32227 raft_consensus.cc:2454] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:51.164650 31900 leader_election.cc:302] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:51.165338 757 raft_consensus.cc:2781] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.165616 757 raft_consensus.cc:686] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:51.165910 757 consensus_queue.cc:237] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.172695 1105 ts_tablet_manager.cc:902] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:51.174100 1078 tablet_bootstrap.cc:492] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:51.174304 1078 ts_tablet_manager.cc:1367] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240501 14:00:51.175398 1078 raft_consensus.cc:348] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.175638 1078 raft_consensus.cc:374] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.175770 1078 raft_consensus.cc:729] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.176054 1078 consensus_queue.cc:260] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.176609 1078 ts_tablet_manager.cc:1398] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:00:51.179497 29096 catalog_manager.cc:5484] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:51.180331 757 raft_consensus.cc:482] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.180495 757 raft_consensus.cc:504] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.182029 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127424670b7549ecab23639640d4e605" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:51.182266 29635 raft_consensus.cc:2454] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:51.182425 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127424670b7549ecab23639640d4e605" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:51.182700 32096 raft_consensus.cc:2454] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:51.183300 31900 leader_election.cc:302] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:51.183574 962 raft_consensus.cc:2781] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.183662 962 raft_consensus.cc:482] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.183748 962 raft_consensus.cc:3037] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.186923 962 raft_consensus.cc:504] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.188802 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127424670b7549ecab23639640d4e605" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:51.188990 29635 raft_consensus.cc:3037] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.191704 29635 raft_consensus.cc:2454] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:51.192133 31898 leader_election.cc:302] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:51.194509 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127424670b7549ecab23639640d4e605" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:51.194721 32096 raft_consensus.cc:3037] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.198839 32096 raft_consensus.cc:2454] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:51.199368 962 leader_election.cc:288] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.199635 962 raft_consensus.cc:2781] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.199827 962 raft_consensus.cc:686] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:51.200114 962 consensus_queue.cc:237] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:51.203857 757 leader_election.cc:288] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.213665 1105 tablet_copy_client.cc:308] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:00:51.214349 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 70ffd2100f104b339bfdc2d272ba3729 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:41457)
I20240501 14:00:51.214558 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 70ffd2100f104b339bfdc2d272ba3729 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:41457: session id = 6a446ac802a7457d9f27076781f21d79-70ffd2100f104b339bfdc2d272ba3729
I20240501 14:00:51.214676 1117 ts_tablet_manager.cc:902] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:51.215991 29096 catalog_manager.cc:5484] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:51.222201 29527 tablet_copy_source_session.cc:205] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.222550 29244 consensus_queue.cc:237] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.223819 1105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffd2100f104b339bfdc2d272ba3729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.232185 32362 raft_consensus.cc:1261] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.232427 32096 raft_consensus.cc:1261] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.233049 1112 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:51.234155 1112 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:00:51.238039 29181 consensus_peers.cc:485] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a412c988fd946d7b7d51606be01d1fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:51.242908 1112 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:51.244510 1105 tablet_copy_client.cc:783] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.246469 1105 tablet_copy_client.cc:647] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.246194 32362 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:51.248715 1105 tablet_copy_client.cc:523] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.254491 1124 ts_tablet_manager.cc:902] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:51.254127 32097 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:51.255105 1105 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:51.262355 29096 catalog_manager.cc:5484] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:51.264230 1117 tablet_copy_client.cc:308] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:00:51.265152 1124 tablet_copy_client.cc:308] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:00:51.265880 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet e5e2a1f3418c477aa65b77bfa171a5a1 from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:59997)
I20240501 14:00:51.266043 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet e5e2a1f3418c477aa65b77bfa171a5a1 from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:59997: session id = b87d188552494b4590afc37a85b98c35-e5e2a1f3418c477aa65b77bfa171a5a1
I20240501 14:00:51.267771 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet fee3fe52d2ae462a823d3ac03f2ce8bc from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:47475)
I20240501 14:00:51.268046 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet fee3fe52d2ae462a823d3ac03f2ce8bc from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:47475: session id = 522f4ff5e9084cbf8462196533c7d7aa-fee3fe52d2ae462a823d3ac03f2ce8bc
I20240501 14:00:51.269868 29657 tablet_copy_source_session.cc:205] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.271414 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e2a1f3418c477aa65b77bfa171a5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.273648 32117 tablet_copy_source_session.cc:205] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.276373 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3fe52d2ae462a823d3ac03f2ce8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.280468 1124 tablet_copy_client.cc:783] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.280719 1124 tablet_copy_client.cc:647] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.283429 1124 tablet_copy_client.cc:523] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.285862 1117 tablet_copy_client.cc:783] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.286156 1117 tablet_copy_client.cc:647] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.286973 1124 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:51.292867 1117 tablet_copy_client.cc:523] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.294103 1124 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.294669 1124 tablet_bootstrap.cc:492] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:00:51.294989 1124 ts_tablet_manager.cc:1367] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240501 14:00:51.295850 1124 raft_consensus.cc:348] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.296155 1124 raft_consensus.cc:729] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.296536 1124 consensus_queue.cc:260] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.297260 1117 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:51.297312 1124 ts_tablet_manager.cc:1398] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:51.298120 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-e5e2a1f3418c477aa65b77bfa171a5a1 received from {username='slave'} at 127.22.33.66:59997
I20240501 14:00:51.298260 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session b87d188552494b4590afc37a85b98c35-e5e2a1f3418c477aa65b77bfa171a5a1 on tablet e5e2a1f3418c477aa65b77bfa171a5a1 with peer b87d188552494b4590afc37a85b98c35
I20240501 14:00:51.311348 1117 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.311806 1117 tablet_bootstrap.cc:492] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:00:51.312067 1117 ts_tablet_manager.cc:1367] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240501 14:00:51.313077 1117 raft_consensus.cc:348] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.313374 1117 raft_consensus.cc:729] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.313860 1117 consensus_queue.cc:260] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:51.314851 1117 ts_tablet_manager.cc:1398] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:51.314882 1105 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.315308 1105 tablet_bootstrap.cc:492] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:00:51.315582 1105 ts_tablet_manager.cc:1367] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.063s user 0.046s sys 0.008s
I20240501 14:00:51.316102 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-fee3fe52d2ae462a823d3ac03f2ce8bc received from {username='slave'} at 127.22.33.67:47475
I20240501 14:00:51.316334 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-fee3fe52d2ae462a823d3ac03f2ce8bc on tablet fee3fe52d2ae462a823d3ac03f2ce8bc with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:00:51.316254 1105 raft_consensus.cc:348] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:51.316506 1105 raft_consensus.cc:729] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.316804 1105 consensus_queue.cc:260] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:51.317503 1105 ts_tablet_manager.cc:1398] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:51.319311 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-70ffd2100f104b339bfdc2d272ba3729 received from {username='slave'} at 127.22.33.71:41457
I20240501 14:00:51.319447 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-70ffd2100f104b339bfdc2d272ba3729 on tablet 70ffd2100f104b339bfdc2d272ba3729 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:51.325345 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:51.333124 32097 raft_consensus.cc:1261] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.333225 29637 raft_consensus.cc:1261] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.333890 1111 consensus_queue.cc:1035] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.334409 1111 consensus_queue.cc:1035] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.338944 1149 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.339685 1149 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:51.342976 29102 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]). Expected version 1 got 0
I20240501 14:00:51.350545 29102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3] alter complete (version 1)
I20240501 14:00:51.360951 29102 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]). Expected version 1 got 0
I20240501 14:00:51.369385 1152 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.370198 1152 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:51.371682 1150 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.371861 29102 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]). Expected version 1 got 0
I20240501 14:00:51.372474 29105 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3]). Expected version 1 got 0
I20240501 14:00:51.372678 1150 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:51.385704 1153 ts_tablet_manager.cc:902] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:51.387759 1153 tablet_copy_client.cc:308] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:00:51.388643 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet 820be6be18af4781ace626f239379cff from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:52917)
I20240501 14:00:51.388859 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet 820be6be18af4781ace626f239379cff from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:52917: session id = 9903ed35bca144bcb74e9bf5102117e7-820be6be18af4781ace626f239379cff
I20240501 14:00:51.391443 32248 tablet_copy_source_session.cc:205] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.393172 1153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820be6be18af4781ace626f239379cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.399374 1153 tablet_copy_client.cc:783] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.399673 1153 tablet_copy_client.cc:647] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.401715 1153 tablet_copy_client.cc:523] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.405128 1153 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:51.411857 1153 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.412231 1153 tablet_bootstrap.cc:492] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:00:51.412870 1153 ts_tablet_manager.cc:1367] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240501 14:00:51.413743 1153 raft_consensus.cc:348] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.414100 1153 raft_consensus.cc:729] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.414458 1153 consensus_queue.cc:260] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.416107 1153 ts_tablet_manager.cc:1398] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:51.417084 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-820be6be18af4781ace626f239379cff received from {username='slave'} at 127.22.33.69:52917
I20240501 14:00:51.417277 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-820be6be18af4781ace626f239379cff on tablet 820be6be18af4781ace626f239379cff with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:51.458088 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:51.467527 1149 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.468058 1149 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:51.468219 1150 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.468672 1150 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:51.470662 1152 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.471132 1152 tablet.cc:1620] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:51.484426 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=6f0a0e9ffa7847a29e5492b2369c56b3] alter complete (version 2)
I20240501 14:00:51.526048 501 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.544817 750 consensus_queue.cc:1035] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.549532 1002 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.571228 949 consensus_queue.cc:1035] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.589393 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:51.597249 1149 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.599953 1152 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.603586 1150 tablet.cc:1722] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.614321 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=6f0a0e9ffa7847a29e5492b2369c56b3] alter complete (version 3)
I20240501 14:00:51.631196 32077 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:51.662456 29226 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:51.685741 29616 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:51.685907 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:51.689375 32207 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:51.696710 29356 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:51.701486 32341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:51.714640 31945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:51.727758 29104 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:51.728036 29104 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:51.730546 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:51.735013 29104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 127424670b7549ecab23639640d4e605
I20240501 14:00:51.734975 962 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.735957 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 127424670b7549ecab23639640d4e605 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:51.736491 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 127424670b7549ecab23639640d4e605 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:51.736789 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 127424670b7549ecab23639640d4e605 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:51.738783 29377 raft_consensus.cc:1203] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:51.758916 32207 tablet_service.cc:1467] Processing CreateTablet for tablet fa4a1cdf1c214e929b1d81ae3df585d9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.759516 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4a1cdf1c214e929b1d81ae3df585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.762194 32228 raft_consensus.cc:1203] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:51.763324 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 12f0264161ec46af90b404aa18da2b62 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.763769 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f0264161ec46af90b404aa18da2b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.765084 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 12f0264161ec46af90b404aa18da2b62 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.765520 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f0264161ec46af90b404aa18da2b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.767007 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 12f0264161ec46af90b404aa18da2b62 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.767273 1124 ts_tablet_manager.cc:902] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:51.767439 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f0264161ec46af90b404aa18da2b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.778333 32075 tablet_service.cc:1467] Processing CreateTablet for tablet fa4a1cdf1c214e929b1d81ae3df585d9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.778815 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4a1cdf1c214e929b1d81ae3df585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.779019 31944 tablet_service.cc:1467] Processing CreateTablet for tablet fa4a1cdf1c214e929b1d81ae3df585d9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.779383 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4a1cdf1c214e929b1d81ae3df585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.782429 32074 tablet_service.cc:1467] Processing CreateTablet for tablet 1a025068d40c4b388421c5e6930847c2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.782974 32074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a025068d40c4b388421c5e6930847c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.783057 962 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.789203 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 1a025068d40c4b388421c5e6930847c2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.789649 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a025068d40c4b388421c5e6930847c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.803830 1206 tablet_replica.cc:330] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:51.804137 1206 raft_consensus.cc:2229] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:51.804399 1206 raft_consensus.cc:2258] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.812129 1214 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:51.812788 1207 tablet_replica.cc:330] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:51.813367 1208 tablet_replica.cc:330] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:51.813655 1207 raft_consensus.cc:2229] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:51.813663 1208 raft_consensus.cc:2229] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:51.813890 1208 raft_consensus.cc:2258] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.814100 1207 raft_consensus.cc:2258] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.814810 1215 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:51.821249 29505 raft_consensus.cc:1203] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:51.825354 1207 ts_tablet_manager.cc:1875] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.831889 1207 ts_tablet_manager.cc:1888] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:51.837473 29104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:51.832948 1212 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:51.844269 1208 ts_tablet_manager.cc:1875] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.845880 1206 ts_tablet_manager.cc:1875] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.847281 29616 tablet_service.cc:1467] Processing CreateTablet for tablet d93c7a889ed248659087d7469a2ada22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]), partition=
I20240501 14:00:51.847635 32341 tablet_service.cc:1467] Processing CreateTablet for tablet d93c7a889ed248659087d7469a2ada22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]), partition=
I20240501 14:00:51.847731 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c7a889ed248659087d7469a2ada22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.847955 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c7a889ed248659087d7469a2ada22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.853127 1211 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:51.853346 29356 tablet_service.cc:1467] Processing CreateTablet for tablet d93c7a889ed248659087d7469a2ada22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]), partition=
I20240501 14:00:51.853703 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c7a889ed248659087d7469a2ada22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.856063 1124 tablet_copy_client.cc:308] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:00:51.856752 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 1a412c988fd946d7b7d51606be01d1fa from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:52231)
I20240501 14:00:51.856957 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 1a412c988fd946d7b7d51606be01d1fa from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:52231: session id = b87d188552494b4590afc37a85b98c35-1a412c988fd946d7b7d51606be01d1fa
I20240501 14:00:51.858515 1211 tablet_bootstrap.cc:654] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.852268 1212 tablet_bootstrap.cc:654] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.860090 29266 tablet_copy_source_session.cc:205] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.863121 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 1a025068d40c4b388421c5e6930847c2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.869067 1215 tablet_bootstrap.cc:654] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.871364 1207 log.cc:1196] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/127424670b7549ecab23639640d4e605
I20240501 14:00:51.871851 1207 ts_tablet_manager.cc:1909] T 127424670b7549ecab23639640d4e605 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:51.868274 1214 tablet_bootstrap.cc:654] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.874433 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table_new_name [id=6f0a0e9ffa7847a29e5492b2369c56b3]) successfully deleted
I20240501 14:00:51.875656 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a025068d40c4b388421c5e6930847c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.886322 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a412c988fd946d7b7d51606be01d1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.893427 1124 tablet_copy_client.cc:783] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.893703 1124 tablet_copy_client.cc:647] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.897929 1124 tablet_copy_client.cc:523] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.904729 1124 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:51.905205 1221 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:51.913086 1212 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:51.913336 1212 ts_tablet_manager.cc:1367] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.003s
I20240501 14:00:51.914310 1212 raft_consensus.cc:348] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.914570 1212 raft_consensus.cc:374] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.914657 1212 raft_consensus.cc:729] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.914877 1212 consensus_queue.cc:260] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.915498 1212 ts_tablet_manager.cc:1398] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:51.915629 1214 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:51.915763 1214 ts_tablet_manager.cc:1367] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.000s
I20240501 14:00:51.915791 1212 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:51.916501 1214 raft_consensus.cc:348] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.917351 1222 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:51.919759 1214 raft_consensus.cc:374] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.919926 1214 raft_consensus.cc:729] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.920365 1223 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:51.920240 1214 consensus_queue.cc:260] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.922047 1223 tablet_bootstrap.cc:654] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.922729 1214 ts_tablet_manager.cc:1398] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:51.923089 1214 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:51.931337 1214 tablet_bootstrap.cc:654] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.938395 1221 tablet_bootstrap.cc:654] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.942206 1215 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:51.942490 1215 ts_tablet_manager.cc:1367] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.128s user 0.005s sys 0.000s
I20240501 14:00:51.943790 1215 raft_consensus.cc:348] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:51.944186 1215 raft_consensus.cc:374] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.944358 1215 raft_consensus.cc:729] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.946781 1215 consensus_queue.cc:260] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:51.955430 1215 ts_tablet_manager.cc:1398] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:00:51.961190 1222 tablet_bootstrap.cc:654] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.964777 1124 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.965337 1124 tablet_bootstrap.cc:492] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:00:51.965677 1124 ts_tablet_manager.cc:1367] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.003s
I20240501 14:00:51.966698 1124 raft_consensus.cc:348] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.967013 1124 raft_consensus.cc:729] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.979499 1221 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:51.979867 1221 ts_tablet_manager.cc:1367] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20240501 14:00:51.980852 1221 raft_consensus.cc:348] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.981249 1221 raft_consensus.cc:374] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.981402 1221 raft_consensus.cc:729] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.981536 1211 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:51.981740 1211 ts_tablet_manager.cc:1367] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.129s user 0.042s sys 0.041s
I20240501 14:00:51.981742 1221 consensus_queue.cc:260] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.981186 1124 consensus_queue.cc:260] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.982620 1221 ts_tablet_manager.cc:1398] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:51.983806 1124 ts_tablet_manager.cc:1398] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20240501 14:00:51.984793 1211 raft_consensus.cc:348] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.985054 1211 raft_consensus.cc:374] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.985153 1211 raft_consensus.cc:729] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.936889 1208 ts_tablet_manager.cc:1888] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:51.985795 1208 log.cc:1196] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/127424670b7549ecab23639640d4e605
I20240501 14:00:51.986310 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-1a412c988fd946d7b7d51606be01d1fa received from {username='slave'} at 127.22.33.66:52231
I20240501 14:00:51.986444 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session b87d188552494b4590afc37a85b98c35-1a412c988fd946d7b7d51606be01d1fa on tablet 1a412c988fd946d7b7d51606be01d1fa with peer b87d188552494b4590afc37a85b98c35
I20240501 14:00:51.986519 1208 ts_tablet_manager.cc:1909] T 127424670b7549ecab23639640d4e605 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:51.987901 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table_new_name [id=6f0a0e9ffa7847a29e5492b2369c56b3]) successfully deleted
I20240501 14:00:51.987390 1211 consensus_queue.cc:260] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:51.989331 1211 ts_tablet_manager.cc:1398] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:51.991313 1211 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:51.995118 1211 tablet_bootstrap.cc:654] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.997186 1222 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:51.997476 1222 ts_tablet_manager.cc:1367] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.003s
I20240501 14:00:51.998507 1222 raft_consensus.cc:348] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:51.998782 1222 raft_consensus.cc:374] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.998893 1222 raft_consensus.cc:729] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.000084 1222 consensus_queue.cc:260] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.000780 1222 ts_tablet_manager.cc:1398] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:52.001708 1206 ts_tablet_manager.cc:1888] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:52.001844 1206 log.cc:1196] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/127424670b7549ecab23639640d4e605
I20240501 14:00:52.002324 1206 ts_tablet_manager.cc:1909] T 127424670b7549ecab23639640d4e605 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:51.984761 1212 tablet_bootstrap.cc:654] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.003435 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 127424670b7549ecab23639640d4e605 (table rebalancer_dynamic_table_new_name [id=6f0a0e9ffa7847a29e5492b2369c56b3]) successfully deleted
I20240501 14:00:52.005806 1223 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:52.006006 1223 ts_tablet_manager.cc:1367] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.000s
I20240501 14:00:52.007411 1223 raft_consensus.cc:348] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.007647 1223 raft_consensus.cc:374] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.007784 1223 raft_consensus.cc:729] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.008002 1223 consensus_queue.cc:260] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.008419 1223 ts_tablet_manager.cc:1398] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:52.009706 1214 tablet_bootstrap.cc:492] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:52.009930 1214 ts_tablet_manager.cc:1367] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.048s
I20240501 14:00:52.011065 1214 raft_consensus.cc:348] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.011379 31965 raft_consensus.cc:1203] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:52.012524 1221 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:52.011371 1214 raft_consensus.cc:374] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.013778 1214 raft_consensus.cc:729] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.014086 1214 consensus_queue.cc:260] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.014743 1214 ts_tablet_manager.cc:1398] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:52.016256 1221 tablet_bootstrap.cc:654] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.018651 1212 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:52.018924 1212 ts_tablet_manager.cc:1367] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240501 14:00:52.020319 1212 raft_consensus.cc:348] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.020601 1212 raft_consensus.cc:374] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.020740 1212 raft_consensus.cc:729] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.021199 1212 consensus_queue.cc:260] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.023849 1212 ts_tablet_manager.cc:1398] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:00:52.025734 1212 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:52.027861 1212 tablet_bootstrap.cc:654] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.032697 1221 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:52.032898 1221 ts_tablet_manager.cc:1367] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240501 14:00:52.037222 1221 raft_consensus.cc:348] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.038089 1221 raft_consensus.cc:374] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.039327 1212 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:52.039525 1212 ts_tablet_manager.cc:1367] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240501 14:00:52.038349 1221 raft_consensus.cc:729] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.039886 1221 consensus_queue.cc:260] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.040477 1212 raft_consensus.cc:348] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.041119 1221 ts_tablet_manager.cc:1398] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:52.046514 1212 raft_consensus.cc:374] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.046702 1212 raft_consensus.cc:729] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.047000 1212 consensus_queue.cc:260] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.049973 1211 tablet_bootstrap.cc:492] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:52.050148 1211 ts_tablet_manager.cc:1367] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.002s
I20240501 14:00:52.050521 1212 ts_tablet_manager.cc:1398] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240501 14:00:52.050905 1211 raft_consensus.cc:348] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.051127 1211 raft_consensus.cc:374] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.051198 1211 raft_consensus.cc:729] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.051426 1211 consensus_queue.cc:260] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.051942 1211 ts_tablet_manager.cc:1398] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:52.080586 1112 raft_consensus.cc:482] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.080807 1112 raft_consensus.cc:504] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.081583 1112 leader_election.cc:288] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:52.082192 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:52.082170 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:52.082422 32227 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:52.082448 32096 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:52.082934 29181 leader_election.cc:302] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:52.083248 1112 raft_consensus.cc:2781] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.083349 1112 raft_consensus.cc:482] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.083488 1112 raft_consensus.cc:3037] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.086015 1112 raft_consensus.cc:504] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.086546 1112 leader_election.cc:288] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:52.087225 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:52.087428 32227 raft_consensus.cc:3037] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.087464 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:52.087661 32096 raft_consensus.cc:3037] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.090387 32227 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:52.090430 32096 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:52.090828 29180 leader_election.cc:302] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:52.091470 1122 raft_consensus.cc:2781] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.091617 1122 raft_consensus.cc:686] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:52.091933 1122 consensus_queue.cc:237] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:52.117940 1111 raft_consensus.cc:482] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.118142 1111 raft_consensus.cc:504] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.118839 1111 leader_election.cc:288] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:52.119755 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4a1cdf1c214e929b1d81ae3df585d9" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:52.120074 32096 raft_consensus.cc:2454] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:52.120493 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4a1cdf1c214e929b1d81ae3df585d9" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:52.120826 32227 raft_consensus.cc:2454] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:52.121628 31900 leader_election.cc:302] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:52.121977 1111 raft_consensus.cc:2781] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.122095 1111 raft_consensus.cc:482] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.122186 1111 raft_consensus.cc:3037] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.126928 1111 raft_consensus.cc:504] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.128327 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4a1cdf1c214e929b1d81ae3df585d9" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:52.128609 32227 raft_consensus.cc:3037] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.129873 963 raft_consensus.cc:482] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.130120 963 raft_consensus.cc:504] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.131201 963 leader_election.cc:288] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:52.132694 32227 raft_consensus.cc:2454] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:52.133430 31899 leader_election.cc:302] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:52.134289 962 raft_consensus.cc:2781] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.136140 1111 leader_election.cc:288] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:52.136189 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4a1cdf1c214e929b1d81ae3df585d9" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:52.136454 32096 raft_consensus.cc:3037] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.140130 32096 raft_consensus.cc:2454] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
W20240501 14:00:52.143643 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 51285.8us
I20240501 14:00:52.148010 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f0264161ec46af90b404aa18da2b62" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:52.148281 29635 raft_consensus.cc:2454] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:52.132894 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f0264161ec46af90b404aa18da2b62" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:52.150451 31965 raft_consensus.cc:2454] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:52.151060 32032 leader_election.cc:302] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:52.155637 963 raft_consensus.cc:2781] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.155802 963 raft_consensus.cc:482] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.155900 963 raft_consensus.cc:3037] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.156914 962 raft_consensus.cc:686] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:52.157251 962 consensus_queue.cc:237] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.158944 963 raft_consensus.cc:504] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.158849 29102 catalog_manager.cc:5484] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:52.160413 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f0264161ec46af90b404aa18da2b62" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:52.160686 29637 raft_consensus.cc:3037] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.164263 29637 raft_consensus.cc:2454] T 12f0264161ec46af90b404aa18da2b62 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:52.164840 32030 leader_election.cc:302] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:52.167918 963 leader_election.cc:288] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:52.168304 963 raft_consensus.cc:2781] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.168368 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f0264161ec46af90b404aa18da2b62" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:52.168565 31965 raft_consensus.cc:3037] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.168691 963 raft_consensus.cc:686] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:52.169024 963 consensus_queue.cc:237] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:52.171128 31965 raft_consensus.cc:2454] T 12f0264161ec46af90b404aa18da2b62 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:52.171116 29102 catalog_manager.cc:5484] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:52.174098 1141 raft_consensus.cc:482] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.174288 1141 raft_consensus.cc:504] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.175166 1141 leader_election.cc:288] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:52.175638 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c7a889ed248659087d7469a2ada22" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:52.175904 32362 raft_consensus.cc:2454] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:52.175978 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c7a889ed248659087d7469a2ada22" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:52.176239 29637 raft_consensus.cc:2454] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:52.176508 29311 leader_election.cc:302] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:52.176823 1141 raft_consensus.cc:2781] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.176954 1141 raft_consensus.cc:482] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.177074 1141 raft_consensus.cc:3037] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.180606 1141 raft_consensus.cc:504] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.182157 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c7a889ed248659087d7469a2ada22" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:52.182320 1141 leader_election.cc:288] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:52.182466 29635 raft_consensus.cc:3037] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.185699 29635 raft_consensus.cc:2454] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:52.186249 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c7a889ed248659087d7469a2ada22" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:52.186475 32362 raft_consensus.cc:3037] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.191704 29310 leader_election.cc:302] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:52.192327 637 raft_consensus.cc:2781] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.192642 32362 raft_consensus.cc:2454] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:52.201835 29102 catalog_manager.cc:5484] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:52.214007 637 raft_consensus.cc:686] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:52.214452 637 consensus_queue.cc:237] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.218734 29102 catalog_manager.cc:5484] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:52.290803 29377 raft_consensus.cc:1203] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:52.375375 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:52.400790 32077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:52.414284 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:52.436908 29635 raft_consensus.cc:1261] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:52.437992 637 consensus_queue.cc:1035] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.440846 32362 raft_consensus.cc:1261] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:52.441737 1249 consensus_queue.cc:1035] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.456197 32207 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:52.457785 29226 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:52.502334 31945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:52.551353 29096 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]). Expected version 1 got 0
I20240501 14:00:52.562300 962 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.575963 29096 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]). Expected version 1 got 0
I20240501 14:00:52.603972 1122 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.627912 1245 consensus_queue.cc:1035] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.634317 963 consensus_queue.cc:1035] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.649338 1243 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.685791 1303 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.686372 1303 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:00:52.691249 1308 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.691989 1308 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:52.695175 1304 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.695349 1242 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.695902 1304 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:52.706073 29096 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]). Expected version 1 got 0
I20240501 14:00:52.706869 32341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:52.708576 29096 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525]). Expected version 1 got 0
I20240501 14:00:52.716995 29356 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:52.723060 29616 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:52.760985 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525] alter complete (version 1)
I20240501 14:00:52.769307 29104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:52.781198 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 67623891d415411a9da7d7ae9523f69a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.781342 29357 tablet_service.cc:1467] Processing CreateTablet for tablet b12b60d5bf304c28ae2041629b8d5bdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.781668 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67623891d415411a9da7d7ae9523f69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.781776 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12b60d5bf304c28ae2041629b8d5bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.781941 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.782337 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.784859 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.785319 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.785468 31945 tablet_service.cc:1467] Processing CreateTablet for tablet b12b60d5bf304c28ae2041629b8d5bdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.785851 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12b60d5bf304c28ae2041629b8d5bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.787727 1321 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:52.788040 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.789331 32340 tablet_service.cc:1467] Processing CreateTablet for tablet b12b60d5bf304c28ae2041629b8d5bdd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.789757 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12b60d5bf304c28ae2041629b8d5bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.790040 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.790184 1321 tablet_bootstrap.cc:654] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.796504 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 67623891d415411a9da7d7ae9523f69a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.796947 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67623891d415411a9da7d7ae9523f69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.803488 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 67623891d415411a9da7d7ae9523f69a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.803951 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67623891d415411a9da7d7ae9523f69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.814148 1324 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:52.815966 1324 tablet_bootstrap.cc:654] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.829775 1325 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:52.838335 1326 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:52.839794 1324 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:52.840109 1325 tablet_bootstrap.cc:654] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.840467 1324 ts_tablet_manager.cc:1367] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 14:00:52.842582 1323 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:52.842167 1324 raft_consensus.cc:348] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.843097 1324 raft_consensus.cc:374] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.843255 1324 raft_consensus.cc:729] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.843602 1324 consensus_queue.cc:260] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.844398 1324 ts_tablet_manager.cc:1398] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:52.844897 1324 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:52.846966 1324 tablet_bootstrap.cc:654] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.847756 1326 tablet_bootstrap.cc:654] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.848083 1321 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:52.848304 1321 ts_tablet_manager.cc:1367] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.020s
I20240501 14:00:52.849224 1321 raft_consensus.cc:348] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.849480 1321 raft_consensus.cc:374] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.849628 1321 raft_consensus.cc:729] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.849917 1321 consensus_queue.cc:260] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.853085 1321 ts_tablet_manager.cc:1398] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:52.854859 1323 tablet_bootstrap.cc:654] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.856628 1324 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:52.856807 1324 ts_tablet_manager.cc:1367] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 14:00:52.857620 1324 raft_consensus.cc:348] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.857905 1324 raft_consensus.cc:374] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.858014 1324 raft_consensus.cc:729] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.858299 1324 consensus_queue.cc:260] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.858403 1325 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:52.858515 1325 ts_tablet_manager.cc:1367] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240501 14:00:52.859026 1324 ts_tablet_manager.cc:1398] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:52.860033 1325 raft_consensus.cc:348] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.860245 1325 raft_consensus.cc:374] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.860356 1325 raft_consensus.cc:729] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.860587 1325 consensus_queue.cc:260] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.861183 1325 ts_tablet_manager.cc:1398] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:52.863071 1323 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:52.863253 1323 ts_tablet_manager.cc:1367] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 14:00:52.863932 1312 raft_consensus.cc:482] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.864058 1312 raft_consensus.cc:504] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.864713 1312 leader_election.cc:288] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:52.865309 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:52.865540 29376 raft_consensus.cc:2454] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:52.866103 32032 leader_election.cc:302] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:52.866798 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:00:52.867517 32031 leader_election.cc:341] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.869230 1312 raft_consensus.cc:2781] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.869340 1312 raft_consensus.cc:482] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.869424 1312 raft_consensus.cc:3037] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.873051 1312 raft_consensus.cc:504] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.873534 1323 raft_consensus.cc:348] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.873814 1323 raft_consensus.cc:374] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.873935 1323 raft_consensus.cc:729] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.874040 1312 leader_election.cc:288] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:52.874083 1326 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:52.874342 1326 ts_tablet_manager.cc:1367] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240501 14:00:52.874274 1323 consensus_queue.cc:260] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.874655 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:52.874853 1323 ts_tablet_manager.cc:1398] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:00:52.875159 1323 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:52.875435 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:52.875416 1326 raft_consensus.cc:348] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.875609 1326 raft_consensus.cc:374] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.875653 29376 raft_consensus.cc:3037] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.876965 1323 tablet_bootstrap.cc:654] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:52.877259 32031 leader_election.cc:341] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.878139 1326 raft_consensus.cc:729] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.878499 1326 consensus_queue.cc:260] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.879339 29376 raft_consensus.cc:2454] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:52.879947 32032 leader_election.cc:302] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:52.887060 1312 raft_consensus.cc:2781] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.887845 1312 raft_consensus.cc:686] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:52.888439 1326 ts_tablet_manager.cc:1398] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:00:52.893337 1326 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:52.894847 1312 consensus_queue.cc:237] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.896397 1326 tablet_bootstrap.cc:654] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.900722 29104 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:52.907428 1326 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:52.914186 1303 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.914724 1303 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:00:52.915189 1304 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.915638 1304 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:52.915832 1326 ts_tablet_manager.cc:1367] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240501 14:00:52.916672 1326 raft_consensus.cc:348] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.916903 1326 raft_consensus.cc:374] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.916999 1326 raft_consensus.cc:729] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.917276 1326 consensus_queue.cc:260] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:52.922628 1326 ts_tablet_manager.cc:1398] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:00:52.930634 29102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f65f9780c1154bfbbe37b118790ad525] alter complete (version 2)
I20240501 14:00:52.932372 1326 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:52.934020 1326 tablet_bootstrap.cc:654] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.935950 1308 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.936468 1308 tablet.cc:1620] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:52.941288 1326 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:52.941485 1326 ts_tablet_manager.cc:1367] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240501 14:00:52.942353 1326 raft_consensus.cc:348] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.942617 1326 raft_consensus.cc:374] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.942806 1326 raft_consensus.cc:729] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.943094 1326 consensus_queue.cc:260] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.945214 1326 ts_tablet_manager.cc:1398] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:52.950215 1070 raft_consensus.cc:482] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.950184 29096 catalog_manager.cc:5484] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:52.950393 1070 raft_consensus.cc:504] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.951182 1070 leader_election.cc:288] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:52.952368 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12b60d5bf304c28ae2041629b8d5bdd" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:52.952600 31964 raft_consensus.cc:2454] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:52.953678 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12b60d5bf304c28ae2041629b8d5bdd" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:52.953945 32296 leader_election.cc:302] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:52.953974 29376 raft_consensus.cc:2454] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:52.956938 1070 raft_consensus.cc:2781] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.957077 1070 raft_consensus.cc:482] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.957180 1070 raft_consensus.cc:3037] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.958230 1323 tablet_bootstrap.cc:492] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:52.958387 1323 ts_tablet_manager.cc:1367] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.036s
I20240501 14:00:52.959093 1323 raft_consensus.cc:348] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.959293 1323 raft_consensus.cc:374] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.959357 1323 raft_consensus.cc:729] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.959606 1323 consensus_queue.cc:260] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.959854 1070 raft_consensus.cc:504] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.960233 1323 ts_tablet_manager.cc:1398] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:52.960491 1070 leader_election.cc:288] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:52.961184 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12b60d5bf304c28ae2041629b8d5bdd" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:52.961231 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12b60d5bf304c28ae2041629b8d5bdd" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:52.961390 31964 raft_consensus.cc:3037] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.961390 29376 raft_consensus.cc:3037] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.964167 31964 raft_consensus.cc:2454] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:52.964180 29376 raft_consensus.cc:2454] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:52.964576 32296 leader_election.cc:302] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:52.965137 1070 raft_consensus.cc:2781] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.979280 1070 raft_consensus.cc:686] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:52.979720 1070 consensus_queue.cc:237] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:52.984421 1339 raft_consensus.cc:482] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.984711 1339 raft_consensus.cc:504] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.986862 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67623891d415411a9da7d7ae9523f69a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:52.987150 29244 raft_consensus.cc:2454] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:52.991039 32294 leader_election.cc:302] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:52.991367 1340 raft_consensus.cc:2781] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.991510 1340 raft_consensus.cc:482] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.991652 1340 raft_consensus.cc:3037] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.992759 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67623891d415411a9da7d7ae9523f69a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:52.993148 31964 raft_consensus.cc:2454] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:52.993849 1339 leader_election.cc:288] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:52.997481 1340 raft_consensus.cc:504] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:52.998936 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67623891d415411a9da7d7ae9523f69a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:52.999191 29244 raft_consensus.cc:3037] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.002703 29244 raft_consensus.cc:2454] T 67623891d415411a9da7d7ae9523f69a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.003446 32294 leader_election.cc:302] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:53.004587 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67623891d415411a9da7d7ae9523f69a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:53.004868 31964 raft_consensus.cc:3037] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.004638 29096 catalog_manager.cc:5484] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:53.005890 1070 raft_consensus.cc:2781] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.006057 1070 raft_consensus.cc:686] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:53.006312 1070 consensus_queue.cc:237] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.008369 31964 raft_consensus.cc:2454] T 67623891d415411a9da7d7ae9523f69a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.009552 1340 leader_election.cc:288] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.032439 29096 catalog_manager.cc:5484] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:53.033644 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:53.042471 1303 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.043503 1308 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.055006 29102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=f65f9780c1154bfbbe37b118790ad525] alter complete (version 3)
I20240501 14:00:53.069810 1304 tablet.cc:1722] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.115962 29505 consensus_queue.cc:237] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:53.118165 32227 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:53.118798 1042 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.119567 29376 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:53.120107 1042 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.122318 1042 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:53.126765 32227 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:53.132506 29096 catalog_manager.cc:5484] T 70ffd2100f104b339bfdc2d272ba3729 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:53.137432 29376 raft_consensus.cc:2932] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:53.141989 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ffd2100f104b339bfdc2d272ba3729 with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:53.149574 1362 tablet_replica.cc:330] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:53.149901 1362 raft_consensus.cc:2229] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.150249 1362 raft_consensus.cc:2258] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.151381 1362 ts_tablet_manager.cc:1875] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.159710 29102 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:53.159921 29102 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:53.160564 1362 ts_tablet_manager.cc:1888] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:00:53.160717 1362 log.cc:1196] T 70ffd2100f104b339bfdc2d272ba3729 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/70ffd2100f104b339bfdc2d272ba3729
I20240501 14:00:53.161794 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 70ffd2100f104b339bfdc2d272ba3729 (table rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]) successfully deleted
I20240501 14:00:53.163936 29102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet d93c7a889ed248659087d7469a2ada22
I20240501 14:00:53.164925 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet d93c7a889ed248659087d7469a2ada22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:53.164968 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet d93c7a889ed248659087d7469a2ada22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:53.165210 32340 tablet_service.cc:1514] Processing DeleteTablet for tablet d93c7a889ed248659087d7469a2ada22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:53.165705 1368 tablet_replica.cc:330] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:53.166055 1368 raft_consensus.cc:2229] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.166446 1368 raft_consensus.cc:2258] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.167747 1368 ts_tablet_manager.cc:1875] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.173416 1370 tablet_replica.cc:330] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:53.173710 1370 raft_consensus.cc:2229] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.174018 1370 raft_consensus.cc:2258] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.175037 1370 ts_tablet_manager.cc:1875] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.179169 1368 ts_tablet_manager.cc:1888] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.179347 1368 log.cc:1196] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/d93c7a889ed248659087d7469a2ada22
I20240501 14:00:53.179967 1368 ts_tablet_manager.cc:1909] T d93c7a889ed248659087d7469a2ada22 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:53.181494 1369 tablet_replica.cc:330] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:53.181741 1369 raft_consensus.cc:2229] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:53.181775 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table_new_name [id=f65f9780c1154bfbbe37b118790ad525]) successfully deleted
I20240501 14:00:53.182154 1369 raft_consensus.cc:2258] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.182924 1369 ts_tablet_manager.cc:1875] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.184376 1370 ts_tablet_manager.cc:1888] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.184618 1370 log.cc:1196] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/d93c7a889ed248659087d7469a2ada22
I20240501 14:00:53.185122 1370 ts_tablet_manager.cc:1909] T d93c7a889ed248659087d7469a2ada22 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:53.186408 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table_new_name [id=f65f9780c1154bfbbe37b118790ad525]) successfully deleted
I20240501 14:00:53.188555 1369 ts_tablet_manager.cc:1888] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.188686 1369 log.cc:1196] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/d93c7a889ed248659087d7469a2ada22
I20240501 14:00:53.189138 1369 ts_tablet_manager.cc:1909] T d93c7a889ed248659087d7469a2ada22 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:00:53.190071 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet d93c7a889ed248659087d7469a2ada22 (table rebalancer_dynamic_table_new_name [id=f65f9780c1154bfbbe37b118790ad525]) successfully deleted
I20240501 14:00:53.266108 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:53.279177 32340 tablet_service.cc:1467] Processing CreateTablet for tablet da35497680934d30a64fbe3c222188d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]), partition=
I20240501 14:00:53.279625 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35497680934d30a64fbe3c222188d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.284014 32077 tablet_service.cc:1467] Processing CreateTablet for tablet da35497680934d30a64fbe3c222188d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]), partition=
I20240501 14:00:53.284404 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35497680934d30a64fbe3c222188d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.295939 31945 tablet_service.cc:1467] Processing CreateTablet for tablet da35497680934d30a64fbe3c222188d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]), partition=
I20240501 14:00:53.296360 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35497680934d30a64fbe3c222188d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.302520 1323 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:53.304123 1323 tablet_bootstrap.cc:654] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.305474 1325 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:53.306285 1070 raft_consensus.cc:482] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.306453 1070 raft_consensus.cc:504] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:53.307003 1325 tablet_bootstrap.cc:654] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.307169 1070 leader_election.cc:288] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:53.307998 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:53.308368 29376 raft_consensus.cc:2379] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6edd6a3430d44c38a19c2804e555059f in current term 1: Already voted for candidate 0a697f3a1ee544daa7073814ab845d58 in this term.
I20240501 14:00:53.309049 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff2fc8d230a45daa4e0c8ba4fec422e" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:53.310169 32296 leader_election.cc:302] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35
I20240501 14:00:53.310776 1070 raft_consensus.cc:3037] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.311985 1323 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:53.312096 1323 ts_tablet_manager.cc:1367] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240501 14:00:53.312700 1323 raft_consensus.cc:348] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.312880 1323 raft_consensus.cc:374] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.312944 1323 raft_consensus.cc:729] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.313362 1323 consensus_queue.cc:260] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.313880 1070 raft_consensus.cc:2726] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:53.314755 1326 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:53.315068 1323 ts_tablet_manager.cc:1398] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:53.316316 1326 tablet_bootstrap.cc:654] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.327713 1325 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:53.327924 1325 ts_tablet_manager.cc:1367] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20240501 14:00:53.328778 1325 raft_consensus.cc:348] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.329044 1325 raft_consensus.cc:374] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.329156 1325 raft_consensus.cc:729] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.329401 1325 consensus_queue.cc:260] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.330308 1325 ts_tablet_manager.cc:1398] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:53.338863 1326 tablet_bootstrap.cc:492] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:53.339115 1326 ts_tablet_manager.cc:1367] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20240501 14:00:53.340135 1326 raft_consensus.cc:348] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.340442 1326 raft_consensus.cc:374] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.340538 1326 raft_consensus.cc:729] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.340809 1326 consensus_queue.cc:260] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.341866 1326 ts_tablet_manager.cc:1398] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:53.357111 32077 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:53.373188 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:53.425482 31945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:53.428591 29356 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:53.431027 32207 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:53.436148 29616 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:53.457662 32340 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:53.466331 1312 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.483915 1312 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.487499 29226 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:53.513156 1070 raft_consensus.cc:482] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.513325 1070 raft_consensus.cc:504] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.513906 1070 leader_election.cc:288] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.514413 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:53.514639 32096 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:53.515064 32296 leader_election.cc:302] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:53.515427 1070 raft_consensus.cc:2781] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.515511 1070 raft_consensus.cc:482] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.515597 1070 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.518414 1070 raft_consensus.cc:504] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.519443 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:53.519620 32096 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.521428 1070 leader_election.cc:288] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.522475 32096 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.522995 32296 leader_election.cc:302] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:53.523334 1070 raft_consensus.cc:2781] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.523901 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:53.524245 31964 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:53.525389 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:53.525995 31964 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.529929 31964 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.540244 1070 raft_consensus.cc:686] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:53.540927 1070 consensus_queue.cc:237] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.547180 1428 raft_consensus.cc:482] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.547844 1428 raft_consensus.cc:504] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.548713 1428 leader_election.cc:288] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.549856 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:53.550184 31964 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:53.550678 32032 leader_election.cc:302] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:53.555655 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:53.558393 1431 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.563067 1430 raft_consensus.cc:2781] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:00:53.563192 1430 raft_consensus.cc:482] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.563311 1430 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.563359 1431 consensus_queue.cc:1035] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.570412 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:53.583683 32207 tablet_service.cc:1467] Processing CreateTablet for tablet b5c8bd2ddd7448bd8a2ccde39c08b763 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.584240 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c8bd2ddd7448bd8a2ccde39c08b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.587102 1430 raft_consensus.cc:504] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.588128 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:53.609313 32077 tablet_service.cc:1467] Processing CreateTablet for tablet b5c8bd2ddd7448bd8a2ccde39c08b763 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.609658 29487 tablet_service.cc:1467] Processing CreateTablet for tablet b5c8bd2ddd7448bd8a2ccde39c08b763 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.609838 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c8bd2ddd7448bd8a2ccde39c08b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.610073 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c8bd2ddd7448bd8a2ccde39c08b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.610615 32208 tablet_service.cc:1467] Processing CreateTablet for tablet b5aa71ecef4d419eb36c7777790e6f08 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.611091 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aa71ecef4d419eb36c7777790e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.614568 31945 tablet_service.cc:1467] Processing CreateTablet for tablet b5aa71ecef4d419eb36c7777790e6f08 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.615087 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aa71ecef4d419eb36c7777790e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.619068 1325 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:53.620201 1070 consensus_queue.cc:1035] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.620755 1325 tablet_bootstrap.cc:654] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.623636 1441 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.626866 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.627365 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.643940 1325 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:53.644186 1325 ts_tablet_manager.cc:1367] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.007s
I20240501 14:00:53.601336 1430 leader_election.cc:288] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.646962 32340 tablet_service.cc:1467] Processing CreateTablet for tablet b5aa71ecef4d419eb36c7777790e6f08 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.647472 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aa71ecef4d419eb36c7777790e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.647822 1325 raft_consensus.cc:348] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.648051 1325 raft_consensus.cc:374] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.648147 1325 raft_consensus.cc:729] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.648335 1325 consensus_queue.cc:260] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.648892 1325 ts_tablet_manager.cc:1398] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:53.649514 1442 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:53.656317 1442 tablet_bootstrap.cc:654] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.665410 1442 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:53.666009 1442 ts_tablet_manager.cc:1367] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240501 14:00:53.666982 1442 raft_consensus.cc:348] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.668136 1442 raft_consensus.cc:374] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.668370 1442 raft_consensus.cc:729] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.668790 1442 consensus_queue.cc:260] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.673449 1442 ts_tablet_manager.cc:1398] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:53.685858 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.686478 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.694814 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.695362 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.710876 1326 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:53.713375 1326 tablet_bootstrap.cc:654] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.717115 1450 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:53.719056 1450 tablet_bootstrap.cc:654] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.723320 1323 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:53.725395 1323 tablet_bootstrap.cc:654] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.727581 1323 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:53.727581 1326 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:53.727766 1323 ts_tablet_manager.cc:1367] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:00:53.727788 1326 ts_tablet_manager.cc:1367] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 14:00:53.728601 1323 raft_consensus.cc:348] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.728896 1323 raft_consensus.cc:374] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.728986 1323 raft_consensus.cc:729] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.729300 1323 consensus_queue.cc:260] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.729214 1326 raft_consensus.cc:348] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.729897 1326 raft_consensus.cc:374] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.729928 1323 ts_tablet_manager.cc:1398] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:53.730020 1326 raft_consensus.cc:729] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.730398 1326 consensus_queue.cc:260] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.731150 1326 ts_tablet_manager.cc:1398] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:53.736150 1326 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:53.736564 1450 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:53.736898 1450 ts_tablet_manager.cc:1367] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240501 14:00:53.738183 1450 raft_consensus.cc:348] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.740883 1450 raft_consensus.cc:374] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.741400 1450 raft_consensus.cc:729] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.742092 1450 consensus_queue.cc:260] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.744379 1450 ts_tablet_manager.cc:1398] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:00:53.744935 1326 tablet_bootstrap.cc:654] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.745615 1450 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:53.750347 1450 tablet_bootstrap.cc:654] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.753659 1326 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:53.753890 1326 ts_tablet_manager.cc:1367] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.001s
I20240501 14:00:53.754894 1326 raft_consensus.cc:348] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.755201 1326 raft_consensus.cc:374] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.755297 1326 raft_consensus.cc:729] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.755604 1326 consensus_queue.cc:260] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.756336 1326 ts_tablet_manager.cc:1398] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20240501 14:00:53.758598 32032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 207875us
I20240501 14:00:53.762814 1451 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:53.774364 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35497680934d30a64fbe3c222188d5" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:53.780831 1450 tablet_bootstrap.cc:492] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:53.781139 1450 ts_tablet_manager.cc:1367] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.005s
I20240501 14:00:53.781958 1450 raft_consensus.cc:348] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.782264 1450 raft_consensus.cc:374] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.782377 1450 raft_consensus.cc:729] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.782948 1450 consensus_queue.cc:260] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.783879 31964 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.785993 1450 ts_tablet_manager.cc:1398] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:00:53.787001 1450 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:53.789229 31964 raft_consensus.cc:2454] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:00:53.790820 1450 tablet_bootstrap.cc:654] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.797596 32032 leader_election.cc:302] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:53.797926 1430 raft_consensus.cc:2781] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:53.798101 1430 raft_consensus.cc:686] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:53.798411 1430 consensus_queue.cc:237] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:53.803180 29103 catalog_manager.cc:5484] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:53.806919 1450 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:53.807621 1450 ts_tablet_manager.cc:1367] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240501 14:00:53.808925 1450 raft_consensus.cc:348] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.810065 1450 raft_consensus.cc:374] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.810231 1451 tablet_bootstrap.cc:654] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.810384 1450 raft_consensus.cc:729] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.810884 1450 consensus_queue.cc:260] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.813833 1429 raft_consensus.cc:482] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.814002 1429 raft_consensus.cc:504] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.812976 1450 ts_tablet_manager.cc:1398] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:53.814642 1429 leader_election.cc:288] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.815006 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:53.815248 32227 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:53.815728 32295 leader_election.cc:302] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:53.816301 1429 raft_consensus.cc:2781] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.816416 1429 raft_consensus.cc:482] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.816509 1429 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.819409 1429 raft_consensus.cc:504] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.823506 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:53.823920 31964 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:53.825922 949 raft_consensus.cc:482] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.826069 949 raft_consensus.cc:504] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.826704 949 leader_election.cc:288] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:53.828289 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c8bd2ddd7448bd8a2ccde39c08b763" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:53.828550 29504 raft_consensus.cc:2454] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:53.828603 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:53.828778 32227 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.830808 32162 leader_election.cc:302] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:53.831419 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c8bd2ddd7448bd8a2ccde39c08b763" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:53.831538 32227 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.831672 32096 raft_consensus.cc:2454] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:53.832265 32295 leader_election.cc:302] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:53.832623 1431 raft_consensus.cc:2781] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.832743 1431 raft_consensus.cc:686] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:53.832821 949 raft_consensus.cc:2781] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.832931 949 raft_consensus.cc:482] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.833026 949 raft_consensus.cc:3037] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.833047 1431 consensus_queue.cc:237] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.838343 949 raft_consensus.cc:504] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.839368 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c8bd2ddd7448bd8a2ccde39c08b763" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:53.839599 32096 raft_consensus.cc:3037] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.839777 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:53.839982 31964 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.842701 31964 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:53.843367 1429 leader_election.cc:288] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:53.845105 32096 raft_consensus.cc:2454] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:53.845593 32163 leader_election.cc:302] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:53.845665 29103 catalog_manager.cc:5484] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:53.846083 949 leader_election.cc:288] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:53.846251 949 raft_consensus.cc:2781] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.846354 949 raft_consensus.cc:686] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:53.846612 949 consensus_queue.cc:237] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:53.848688 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c8bd2ddd7448bd8a2ccde39c08b763" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:53.848910 29504 raft_consensus.cc:3037] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.853238 29504 raft_consensus.cc:2454] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:53.867154 1451 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:53.867316 1451 ts_tablet_manager.cc:1367] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.002s
I20240501 14:00:53.867996 1451 raft_consensus.cc:348] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.868211 1451 raft_consensus.cc:374] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.868335 1451 raft_consensus.cc:729] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.868574 1451 consensus_queue.cc:260] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:53.869246 1451 ts_tablet_manager.cc:1398] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:53.902101 29103 catalog_manager.cc:5484] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:53.961669 31964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96"
dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35754
I20240501 14:00:53.961958 31964 raft_consensus.cc:595] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:00:53.982964 1242 raft_consensus.cc:979] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: : Instructing follower 0a697f3a1ee544daa7073814ab845d58 to start an election
I20240501 14:00:53.983177 1242 raft_consensus.cc:1067] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Signalling peer 0a697f3a1ee544daa7073814ab845d58 to start an election
I20240501 14:00:53.984277 32096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96"
dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
from {username='slave'} at 127.22.33.69:40233
I20240501 14:00:53.984525 32096 raft_consensus.cc:482] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:53.984658 32096 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.987149 32096 raft_consensus.cc:504] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.988026 32096 leader_election.cc:288] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:53.988248 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:53.988476 31964 raft_consensus.cc:3032] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:53.988569 31964 raft_consensus.cc:729] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:53.988638 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:53.988860 32227 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.988909 31964 consensus_queue.cc:260] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:53.989414 31964 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.992146 31964 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:00:53.992677 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:53.993083 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414c148db46543b786bfd8b0faecfe96" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:53.993319 32360 raft_consensus.cc:3037] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.994494 32227 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:00:53.995100 32031 leader_election.cc:302] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:53.995694 1434 raft_consensus.cc:2781] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:53.995838 1434 raft_consensus.cc:686] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:53.996166 32360 raft_consensus.cc:2454] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:00:53.996150 1434 consensus_queue.cc:237] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.000787 31964 raft_consensus.cc:1261] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:00:54.001554 1430 consensus_queue.cc:1035] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.001751 32360 raft_consensus.cc:3032] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:54.001869 32360 raft_consensus.cc:729] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:54.002183 32360 consensus_queue.cc:260] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:54.002543 32360 raft_consensus.cc:3037] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.004559 29103 catalog_manager.cc:5484] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 1 to 2, leader changed from 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:54.005419 32360 raft_consensus.cc:1261] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:00:54.005888 1434 consensus_queue.cc:1035] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.006093 1434 consensus_queue.cc:1225] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Peer 6edd6a3430d44c38a19c2804e555059f log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240501 14:00:54.007174 32360 pending_rounds.cc:76] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Aborting all ops after (but not including) 0
I20240501 14:00:54.007328 32360 pending_rounds.cc:97] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240501 14:00:54.007442 32360 raft_consensus.cc:2866] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:00:54.010658 29103 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]). Expected version 1 got 0
I20240501 14:00:54.029762 1479 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.030711 1479 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:54.035218 1429 raft_consensus.cc:482] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.035467 1429 raft_consensus.cc:504] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:54.036464 1429 leader_election.cc:288] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:54.037696 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356722d5ec9d4d3d8ba08b4e7c2b2b11" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:54.038044 29376 raft_consensus.cc:2454] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:54.038431 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356722d5ec9d4d3d8ba08b4e7c2b2b11" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:54.038636 32227 raft_consensus.cc:2454] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:54.039031 32295 leader_election.cc:302] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:54.039521 1431 raft_consensus.cc:2781] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.039644 1431 raft_consensus.cc:482] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.039754 1431 raft_consensus.cc:3037] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.042919 1431 raft_consensus.cc:504] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:54.043886 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356722d5ec9d4d3d8ba08b4e7c2b2b11" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:54.044067 32227 raft_consensus.cc:3037] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.044174 29103 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]). Expected version 1 got 0
I20240501 14:00:54.045991 29099 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]). Expected version 1 got 0
I20240501 14:00:54.046793 1480 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.046945 32227 raft_consensus.cc:2454] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:54.047372 1480 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:54.047333 32295 leader_election.cc:302] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:54.047631 1429 raft_consensus.cc:2781] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.047778 1429 raft_consensus.cc:686] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:54.048069 1429 consensus_queue.cc:237] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:54.048913 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356722d5ec9d4d3d8ba08b4e7c2b2b11" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:00:54.049249 29376 raft_consensus.cc:3037] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.054337 29376 raft_consensus.cc:2454] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:54.055339 1431 leader_election.cc:288] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:00:54.063452 29103 catalog_manager.cc:5484] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:54.070864 1481 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.072139 1481 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:54.074153 29103 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252]). Expected version 1 got 0
I20240501 14:00:54.075224 29098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252] alter complete (version 1)
I20240501 14:00:54.179949 29104 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:54.191637 1479 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.192149 1479 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:54.192932 1480 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.193419 1480 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:54.193951 1481 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.194402 1481 tablet.cc:1620] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:54.206382 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=9e0b4372b20f4b298bf4a94783525252] alter complete (version 2)
I20240501 14:00:54.268726 29226 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:54.268296 32077 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:54.292084 31945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:54.302560 32206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:54.312948 29099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:54.319876 1429 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.323495 1479 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.330677 1480 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.334182 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=9e0b4372b20f4b298bf4a94783525252] alter complete (version 3)
I20240501 14:00:54.334770 1481 tablet.cc:1722] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.347754 1439 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.360504 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:54.365339 29616 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:54.366811 29356 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:54.399001 949 consensus_queue.cc:1035] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.401679 32342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:54.412480 31964 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:00:54.413472 1312 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:54.420621 949 consensus_queue.cc:1035] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.441825 29099 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.442049 29099 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.446197 29099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet da35497680934d30a64fbe3c222188d5
I20240501 14:00:54.447407 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet da35497680934d30a64fbe3c222188d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:54.447480 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet da35497680934d30a64fbe3c222188d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:54.448267 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet da35497680934d30a64fbe3c222188d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:54.487699 1534 tablet_replica.cc:330] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:54.487957 1534 raft_consensus.cc:2229] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.488309 1534 raft_consensus.cc:2258] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.492199 1534 ts_tablet_manager.cc:1875] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.493108 1535 tablet_replica.cc:330] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:54.494618 1535 raft_consensus.cc:2229] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.495771 1535 raft_consensus.cc:2258] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.497566 1535 ts_tablet_manager.cc:1875] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.506827 1534 ts_tablet_manager.cc:1888] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 14:00:54.509268 1534 log.cc:1196] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/da35497680934d30a64fbe3c222188d5
I20240501 14:00:54.510236 1534 ts_tablet_manager.cc:1909] T da35497680934d30a64fbe3c222188d5 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
W20240501 14:00:54.510672 32162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 62072.5us
I20240501 14:00:54.513710 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table_new_name [id=9e0b4372b20f4b298bf4a94783525252]) successfully deleted
I20240501 14:00:54.517489 1533 tablet_replica.cc:330] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:54.517771 1533 raft_consensus.cc:2229] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:00:54.518187 1533 raft_consensus.cc:2258] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.519155 1533 ts_tablet_manager.cc:1875] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.525681 1533 ts_tablet_manager.cc:1888] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 14:00:54.525913 1533 log.cc:1196] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/da35497680934d30a64fbe3c222188d5
I20240501 14:00:54.526006 1535 ts_tablet_manager.cc:1888] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 14:00:54.526204 1535 log.cc:1196] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/da35497680934d30a64fbe3c222188d5
I20240501 14:00:54.526393 1533 ts_tablet_manager.cc:1909] T da35497680934d30a64fbe3c222188d5 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:54.526770 1535 ts_tablet_manager.cc:1909] T da35497680934d30a64fbe3c222188d5 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:54.527834 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table_new_name [id=9e0b4372b20f4b298bf4a94783525252]) successfully deleted
I20240501 14:00:54.528913 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet da35497680934d30a64fbe3c222188d5 (table rebalancer_dynamic_table_new_name [id=9e0b4372b20f4b298bf4a94783525252]) successfully deleted
I20240501 14:00:54.549254 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:54.568049 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]), partition=
I20240501 14:00:54.568498 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2cf3cb72c48688e4714cd90cdf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.568495 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]), partition=
I20240501 14:00:54.568840 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2cf3cb72c48688e4714cd90cdf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.569536 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]), partition=
I20240501 14:00:54.569876 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2cf3cb72c48688e4714cd90cdf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.581091 1541 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:54.582154 1539 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:54.583196 1541 tablet_bootstrap.cc:654] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.584954 1539 tablet_bootstrap.cc:654] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.587122 1540 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:54.589668 1540 tablet_bootstrap.cc:654] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.592191 1541 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:54.592464 1541 ts_tablet_manager.cc:1367] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240501 14:00:54.593622 1541 raft_consensus.cc:348] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.594051 1541 raft_consensus.cc:374] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.594210 1541 raft_consensus.cc:729] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.594552 1541 consensus_queue.cc:260] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.595760 1540 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:54.595927 1540 ts_tablet_manager.cc:1367] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 14:00:54.596325 1541 ts_tablet_manager.cc:1398] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:54.596724 1540 raft_consensus.cc:348] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.596880 1540 raft_consensus.cc:374] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.597183 1540 raft_consensus.cc:729] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.597534 1540 consensus_queue.cc:260] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.598119 1540 ts_tablet_manager.cc:1398] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:54.602334 32228 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:00:54.603081 1312 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:54.612241 32362 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:00:54.616120 1434 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:54.608865 1539 tablet_bootstrap.cc:492] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:54.617283 1539 ts_tablet_manager.cc:1367] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20240501 14:00:54.618470 1539 raft_consensus.cc:348] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.618754 1539 raft_consensus.cc:374] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.618903 1539 raft_consensus.cc:729] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.619153 1439 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.619282 1539 consensus_queue.cc:260] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.620664 1539 ts_tablet_manager.cc:1398] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:54.626399 1431 raft_consensus.cc:482] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.626566 1431 raft_consensus.cc:504] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.627238 1431 leader_election.cc:288] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:54.627914 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2cf3cb72c48688e4714cd90cdf966" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:54.628196 29504 raft_consensus.cc:2454] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:54.628129 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:54.629053 32295 leader_election.cc:302] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:54.630929 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2cf3cb72c48688e4714cd90cdf966" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:54.631148 32227 raft_consensus.cc:2454] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:54.632115 1431 raft_consensus.cc:2781] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.632242 1431 raft_consensus.cc:482] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.632330 1431 raft_consensus.cc:3037] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.641901 1431 raft_consensus.cc:504] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.642448 1431 leader_election.cc:288] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:54.643178 29504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2cf3cb72c48688e4714cd90cdf966" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:54.643404 29504 raft_consensus.cc:3037] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.644623 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2cf3cb72c48688e4714cd90cdf966" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:54.645152 32227 raft_consensus.cc:3037] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.650854 1429 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.650987 29504 raft_consensus.cc:2454] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:54.652010 32295 leader_election.cc:302] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:54.652771 1429 raft_consensus.cc:2781] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.652906 1429 raft_consensus.cc:686] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:54.653178 1429 consensus_queue.cc:237] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.657514 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fc97541db4467b5089bbeaf9bfca0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.658013 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fc97541db4467b5089bbeaf9bfca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.664391 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fc97541db4467b5089bbeaf9bfca0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.665028 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fc97541db4467b5089bbeaf9bfca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.666548 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 9e32147255254f2d8baff46c9b2d0e60 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.667032 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e32147255254f2d8baff46c9b2d0e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.670645 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 9e32147255254f2d8baff46c9b2d0e60 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.671046 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e32147255254f2d8baff46c9b2d0e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.671059 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 59c54188ad9e40dd925b50a250dcea04 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.671417 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c54188ad9e40dd925b50a250dcea04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.672017 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 9e32147255254f2d8baff46c9b2d0e60 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.672416 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e32147255254f2d8baff46c9b2d0e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.673278 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 59c54188ad9e40dd925b50a250dcea04 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.673622 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c54188ad9e40dd925b50a250dcea04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.676482 32227 raft_consensus.cc:2454] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:54.677378 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0fc97541db4467b5089bbeaf9bfca0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.677738 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fc97541db4467b5089bbeaf9bfca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.680109 29097 catalog_manager.cc:5484] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:54.681519 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 59c54188ad9e40dd925b50a250dcea04 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.681924 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c54188ad9e40dd925b50a250dcea04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.686858 1559 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:54.688598 1559 tablet_bootstrap.cc:654] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.689388 1560 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:54.693310 1560 tablet_bootstrap.cc:654] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.713246 1541 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:54.715195 1541 tablet_bootstrap.cc:654] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.716220 1560 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:54.716405 1560 ts_tablet_manager.cc:1367] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240501 14:00:54.717414 1560 raft_consensus.cc:348] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.720036 1560 raft_consensus.cc:374] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.720216 1560 raft_consensus.cc:729] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.721199 1560 consensus_queue.cc:260] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.721905 1560 ts_tablet_manager.cc:1398] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:54.723259 1559 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:54.723407 1559 ts_tablet_manager.cc:1367] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.004s
I20240501 14:00:54.724189 1559 raft_consensus.cc:348] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.724555 1559 raft_consensus.cc:374] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.724727 1559 raft_consensus.cc:729] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.725482 1559 consensus_queue.cc:260] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.726332 1559 ts_tablet_manager.cc:1398] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:54.726778 1559 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:54.728569 1559 tablet_bootstrap.cc:654] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.743736 1541 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:54.744001 1541 ts_tablet_manager.cc:1367] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.002s
I20240501 14:00:54.746142 1541 raft_consensus.cc:348] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.746374 1541 raft_consensus.cc:374] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.746448 1541 raft_consensus.cc:729] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.746685 1541 consensus_queue.cc:260] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.747388 1541 ts_tablet_manager.cc:1398] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:54.747758 1541 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:54.749823 1541 tablet_bootstrap.cc:654] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.751948 1558 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:54.753787 1541 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:54.753963 1541 ts_tablet_manager.cc:1367] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 14:00:54.754086 1558 tablet_bootstrap.cc:654] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.754662 1541 raft_consensus.cc:348] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.754853 1541 raft_consensus.cc:374] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.754990 1541 raft_consensus.cc:729] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.755231 1541 consensus_queue.cc:260] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.755864 1541 ts_tablet_manager.cc:1398] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:54.756150 1541 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:54.758103 1541 tablet_bootstrap.cc:654] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.759166 1559 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:54.761143 1559 ts_tablet_manager.cc:1367] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.014s
I20240501 14:00:54.762367 1559 raft_consensus.cc:348] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.762753 1559 raft_consensus.cc:374] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.762887 1559 raft_consensus.cc:729] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.763243 1559 consensus_queue.cc:260] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.764076 1559 ts_tablet_manager.cc:1398] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:54.764506 1559 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:54.769637 1558 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:54.769866 1558 ts_tablet_manager.cc:1367] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240501 14:00:54.770727 1558 raft_consensus.cc:348] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.770992 1558 raft_consensus.cc:374] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.771144 1558 raft_consensus.cc:729] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.771420 1559 tablet_bootstrap.cc:654] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.771440 1558 consensus_queue.cc:260] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.774070 1558 ts_tablet_manager.cc:1398] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:54.774405 1558 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:54.775249 1541 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:54.775431 1541 ts_tablet_manager.cc:1367] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20240501 14:00:54.775899 1558 tablet_bootstrap.cc:654] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.776170 1541 raft_consensus.cc:348] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.776382 1541 raft_consensus.cc:374] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.776439 1541 raft_consensus.cc:729] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.778079 1541 consensus_queue.cc:260] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.781239 1558 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:54.781389 1558 ts_tablet_manager.cc:1367] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:54.781483 1541 ts_tablet_manager.cc:1398] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:54.782064 1558 raft_consensus.cc:348] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.782298 1558 raft_consensus.cc:374] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.782378 1558 raft_consensus.cc:729] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.782636 1558 consensus_queue.cc:260] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.783202 1558 ts_tablet_manager.cc:1398] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:54.783906 1559 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:54.784040 1559 ts_tablet_manager.cc:1367] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240501 14:00:54.784823 1559 raft_consensus.cc:348] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.785065 1559 raft_consensus.cc:374] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.785156 1559 raft_consensus.cc:729] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.785352 1559 consensus_queue.cc:260] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.785907 1559 ts_tablet_manager.cc:1398] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:54.794381 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:54.800071 29505 raft_consensus.cc:1261] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:54.800144 32227 raft_consensus.cc:1261] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:54.800750 1439 consensus_queue.cc:1035] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.801185 1429 consensus_queue.cc:1035] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.806236 1480 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.806859 1480 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:54.808830 29097 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]). Expected version 1 got 0
I20240501 14:00:54.818202 1581 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.818876 1581 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:54.826771 29100 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]). Expected version 1 got 0
I20240501 14:00:54.827296 29099 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]). Expected version 1 got 0
I20240501 14:00:54.831130 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504] alter complete (version 1)
I20240501 14:00:54.836673 1434 raft_consensus.cc:482] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.836876 1434 raft_consensus.cc:504] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.837615 1434 leader_election.cc:288] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:54.838546 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fc97541db4467b5089bbeaf9bfca0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:54.838922 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fc97541db4467b5089bbeaf9bfca0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:54.838964 29635 raft_consensus.cc:2454] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:54.839144 32362 raft_consensus.cc:2454] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:54.839582 32030 leader_election.cc:302] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:54.839897 1434 raft_consensus.cc:2781] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.840061 1434 raft_consensus.cc:482] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.840164 1434 raft_consensus.cc:3037] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.844202 1434 raft_consensus.cc:504] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.845386 1434 leader_election.cc:288] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:54.845927 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fc97541db4467b5089bbeaf9bfca0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:54.846174 32362 raft_consensus.cc:3037] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.846275 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0fc97541db4467b5089bbeaf9bfca0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:54.846592 29635 raft_consensus.cc:3037] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.850204 29635 raft_consensus.cc:2454] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:54.850715 32030 leader_election.cc:302] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:54.853080 32362 raft_consensus.cc:2454] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:54.853147 1434 raft_consensus.cc:2781] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.853310 1434 raft_consensus.cc:686] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:54.853621 1434 consensus_queue.cc:237] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.857740 29097 catalog_manager.cc:5484] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:54.860718 1586 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.862190 1586 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:00:54.864171 29100 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504]). Expected version 1 got 0
I20240501 14:00:54.925180 962 raft_consensus.cc:482] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.925375 962 raft_consensus.cc:504] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.926162 962 leader_election.cc:288] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:54.926786 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c54188ad9e40dd925b50a250dcea04" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:54.927095 29635 raft_consensus.cc:2454] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:54.927729 31898 leader_election.cc:302] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:54.928908 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c54188ad9e40dd925b50a250dcea04" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:54.929174 32362 raft_consensus.cc:2454] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:54.931993 29099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:54.938881 962 raft_consensus.cc:2781] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.939101 962 raft_consensus.cc:482] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.939218 962 raft_consensus.cc:3037] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.948247 1480 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.948825 1480 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:54.954643 1581 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.955039 1586 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.955325 1581 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:54.955576 1586 tablet.cc:1620] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:00:54.964104 962 raft_consensus.cc:504] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.966250 962 leader_election.cc:288] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:54.967492 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=749ad29d23164735909c364e89dea504] alter complete (version 2)
I20240501 14:00:54.976680 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c54188ad9e40dd925b50a250dcea04" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:54.976930 29635 raft_consensus.cc:3037] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.980779 29635 raft_consensus.cc:2454] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:54.983013 31898 leader_election.cc:302] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:54.983770 962 raft_consensus.cc:2781] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.984078 962 raft_consensus.cc:686] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:54.984519 962 consensus_queue.cc:237] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:54.992362 1002 raft_consensus.cc:482] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.992628 1002 raft_consensus.cc:504] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:54.993739 1002 leader_election.cc:288] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:55.003983 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e32147255254f2d8baff46c9b2d0e60" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:55.004317 32362 raft_consensus.cc:2454] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:55.005627 29571 leader_election.cc:302] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:55.006356 1548 raft_consensus.cc:2781] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.006471 1548 raft_consensus.cc:482] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.006603 1548 raft_consensus.cc:3037] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.016749 1548 raft_consensus.cc:504] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:55.020051 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e32147255254f2d8baff46c9b2d0e60" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:55.020314 32362 raft_consensus.cc:3037] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.023204 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e32147255254f2d8baff46c9b2d0e60" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:55.023452 31964 raft_consensus.cc:2454] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:00:55.023954 32362 raft_consensus.cc:2454] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:55.023977 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e32147255254f2d8baff46c9b2d0e60" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:55.024153 31964 raft_consensus.cc:3037] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.024540 29571 leader_election.cc:302] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:55.024885 1002 raft_consensus.cc:2781] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.025040 1002 raft_consensus.cc:686] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:00:55.025344 1002 consensus_queue.cc:237] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:55.026973 31964 raft_consensus.cc:2454] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:00:55.027454 1548 leader_election.cc:288] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:55.042181 29099 catalog_manager.cc:5484] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:55.049252 29099 catalog_manager.cc:5484] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:55.051743 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c54188ad9e40dd925b50a250dcea04" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:55.051971 32362 raft_consensus.cc:3037] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.054759 32362 raft_consensus.cc:2454] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:55.072511 29099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:55.092453 1480 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.092850 1586 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.097574 1581 tablet.cc:1722] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.103546 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=749ad29d23164735909c364e89dea504] alter complete (version 3)
I20240501 14:00:55.208904 29099 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:55.209229 29099 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:55.215492 29099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 90c2cf3cb72c48688e4714cd90cdf966
I20240501 14:00:55.216483 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:55.216898 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:55.217304 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2cf3cb72c48688e4714cd90cdf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:55.226783 1642 tablet_replica.cc:330] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:55.227020 1642 raft_consensus.cc:2229] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:55.227499 1642 raft_consensus.cc:2258] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.228053 1640 tablet_replica.cc:330] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:55.228340 1640 raft_consensus.cc:2229] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:55.229511 1642 ts_tablet_manager.cc:1875] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.229864 1640 raft_consensus.cc:2258] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.231046 1640 ts_tablet_manager.cc:1875] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.234086 29226 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:55.250586 31944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:55.251909 1640 ts_tablet_manager.cc:1888] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.252084 32077 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:55.254078 1642 ts_tablet_manager.cc:1888] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.254328 1642 log.cc:1196] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/90c2cf3cb72c48688e4714cd90cdf966
I20240501 14:00:55.254941 1642 ts_tablet_manager.cc:1909] T 90c2cf3cb72c48688e4714cd90cdf966 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:55.255125 1640 log.cc:1196] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/90c2cf3cb72c48688e4714cd90cdf966
I20240501 14:00:55.255672 1640 ts_tablet_manager.cc:1909] T 90c2cf3cb72c48688e4714cd90cdf966 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:55.258430 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table_new_name [id=749ad29d23164735909c364e89dea504]) successfully deleted
I20240501 14:00:55.261123 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table_new_name [id=749ad29d23164735909c364e89dea504]) successfully deleted
I20240501 14:00:55.288058 1641 tablet_replica.cc:330] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:55.289405 1641 raft_consensus.cc:2229] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:55.290958 32206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:55.298936 1641 raft_consensus.cc:2258] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.290006 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:55.302127 1641 ts_tablet_manager.cc:1875] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.304404 29356 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:55.326000 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:55.329346 1434 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.339396 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 150338a5de7544e29cade37e0ca8f820 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]), partition=
I20240501 14:00:55.339475 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 150338a5de7544e29cade37e0ca8f820 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]), partition=
I20240501 14:00:55.339840 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150338a5de7544e29cade37e0ca8f820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.339851 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150338a5de7544e29cade37e0ca8f820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.343698 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 150338a5de7544e29cade37e0ca8f820 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]), partition=
I20240501 14:00:55.344062 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150338a5de7544e29cade37e0ca8f820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.353175 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:55.354795 1641 ts_tablet_manager.cc:1888] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.355026 1641 log.cc:1196] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/90c2cf3cb72c48688e4714cd90cdf966
I20240501 14:00:55.355600 1641 ts_tablet_manager.cc:1909] T 90c2cf3cb72c48688e4714cd90cdf966 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:55.358819 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 90c2cf3cb72c48688e4714cd90cdf966 (table rebalancer_dynamic_table_new_name [id=749ad29d23164735909c364e89dea504]) successfully deleted
I20240501 14:00:55.375077 1649 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:55.378690 1649 tablet_bootstrap.cc:654] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.403343 1649 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:55.403628 1649 ts_tablet_manager.cc:1367] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20240501 14:00:55.404502 1649 raft_consensus.cc:348] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.404848 1649 raft_consensus.cc:374] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.405014 1649 raft_consensus.cc:729] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.405292 1649 consensus_queue.cc:260] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.419656 1648 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:55.420200 1647 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:55.422155 1649 ts_tablet_manager.cc:1398] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240501 14:00:55.430286 1650 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.450524 32342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:55.454985 1648 tablet_bootstrap.cc:654] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.456058 1647 tablet_bootstrap.cc:654] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.491401 1647 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:55.491607 1647 ts_tablet_manager.cc:1367] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.000s
I20240501 14:00:55.492565 1647 raft_consensus.cc:348] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.492780 1647 raft_consensus.cc:374] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.492918 1647 raft_consensus.cc:729] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.494534 1647 consensus_queue.cc:260] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.495143 1647 ts_tablet_manager.cc:1398] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:55.503691 1648 tablet_bootstrap.cc:492] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:55.503906 1648 ts_tablet_manager.cc:1367] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.008s
I20240501 14:00:55.504866 1648 raft_consensus.cc:348] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.505147 1648 raft_consensus.cc:374] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.505381 1648 raft_consensus.cc:729] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.505654 1648 consensus_queue.cc:260] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.506255 1648 ts_tablet_manager.cc:1398] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:55.557848 1122 raft_consensus.cc:482] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.558014 1122 raft_consensus.cc:504] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.558713 1122 leader_election.cc:288] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:55.559720 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150338a5de7544e29cade37e0ca8f820" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:55.560034 31964 raft_consensus.cc:2454] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:55.560145 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150338a5de7544e29cade37e0ca8f820" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:55.560360 32097 raft_consensus.cc:2454] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:55.560490 29181 leader_election.cc:302] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:55.561018 1448 raft_consensus.cc:2781] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.561121 1448 raft_consensus.cc:482] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.561229 1448 raft_consensus.cc:3037] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.563920 1448 raft_consensus.cc:504] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.565397 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150338a5de7544e29cade37e0ca8f820" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:55.565609 31964 raft_consensus.cc:3037] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.567986 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150338a5de7544e29cade37e0ca8f820" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:55.568215 32097 raft_consensus.cc:3037] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.568413 31964 raft_consensus.cc:2454] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:55.569097 29181 leader_election.cc:302] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:55.569380 1122 raft_consensus.cc:2781] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.571696 1600 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.576483 32097 raft_consensus.cc:2454] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:55.577246 1448 leader_election.cc:288] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:55.579681 1122 raft_consensus.cc:686] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:55.579969 1122 consensus_queue.cc:237] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:00:55.583910 29097 catalog_manager.cc:5484] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:00:55.584821 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:55.607461 1600 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.610827 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb574dc421745618ba2a11f0b3f9110 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.610939 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb574dc421745618ba2a11f0b3f9110 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.611325 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb574dc421745618ba2a11f0b3f9110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.611366 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb574dc421745618ba2a11f0b3f9110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.613493 32206 tablet_service.cc:1467] Processing CreateTablet for tablet cd7a7501266946dca6bc838d10524c4a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.613934 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd7a7501266946dca6bc838d10524c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.618666 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb574dc421745618ba2a11f0b3f9110 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.619225 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb574dc421745618ba2a11f0b3f9110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.626307 31945 tablet_service.cc:1467] Processing CreateTablet for tablet cd7a7501266946dca6bc838d10524c4a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.626919 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd7a7501266946dca6bc838d10524c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.630224 32342 tablet_service.cc:1467] Processing CreateTablet for tablet cd7a7501266946dca6bc838d10524c4a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.630815 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd7a7501266946dca6bc838d10524c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.631551 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 68f3644410a24fc1a9457ff46a10c180 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.632237 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f3644410a24fc1a9457ff46a10c180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.633157 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 68f3644410a24fc1a9457ff46a10c180 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.633543 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f3644410a24fc1a9457ff46a10c180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.636870 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 68f3644410a24fc1a9457ff46a10c180 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.637310 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f3644410a24fc1a9457ff46a10c180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.641729 1649 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:55.641853 1548 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.643374 1649 tablet_bootstrap.cc:654] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.648159 1002 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.668658 1647 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:55.672189 1647 tablet_bootstrap.cc:654] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.689610 1647 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:55.689913 1647 ts_tablet_manager.cc:1367] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
I20240501 14:00:55.691004 1647 raft_consensus.cc:348] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.691447 1647 raft_consensus.cc:374] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.691627 1647 raft_consensus.cc:729] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.691993 1647 consensus_queue.cc:260] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.694167 1647 ts_tablet_manager.cc:1398] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:55.694577 1647 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:55.704955 1677 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:55.710721 1649 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:55.711045 1649 ts_tablet_manager.cc:1367] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.009s
I20240501 14:00:55.712114 1647 tablet_bootstrap.cc:654] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.712196 29099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:55.719995 1670 raft_consensus.cc:482] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.720197 1670 raft_consensus.cc:504] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.721316 1670 leader_election.cc:288] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:55.723336 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:00:55.723868 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:00:55.724901 31900 leader_election.cc:341] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:55.725124 1677 tablet_bootstrap.cc:654] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:55.725172 31900 leader_election.cc:341] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:55.725301 31900 leader_election.cc:302] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35
I20240501 14:00:55.726262 1670 raft_consensus.cc:2726] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:55.728499 1647 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:55.728926 1647 ts_tablet_manager.cc:1367] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20240501 14:00:55.730264 1649 raft_consensus.cc:348] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.730525 1649 raft_consensus.cc:374] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.730628 1649 raft_consensus.cc:729] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.730885 1649 consensus_queue.cc:260] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.731457 1649 ts_tablet_manager.cc:1398] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240501 14:00:55.731756 1649 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:55.731989 1672 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:55.732770 32097 raft_consensus.cc:1261] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:55.733218 1647 raft_consensus.cc:348] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.733526 1122 consensus_queue.cc:1035] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.733422 1647 raft_consensus.cc:374] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.733844 1647 raft_consensus.cc:729] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.734302 1647 consensus_queue.cc:260] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.735955 1649 tablet_bootstrap.cc:654] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.737668 1673 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:55.739308 31964 raft_consensus.cc:1261] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:55.740109 1660 consensus_queue.cc:1035] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.742241 1647 ts_tablet_manager.cc:1398] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:00:55.744999 1672 tablet_bootstrap.cc:654] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.745627 1692 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.746289 1692 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:55.748958 29103 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]). Expected version 1 got 0
I20240501 14:00:55.750365 1673 tablet_bootstrap.cc:654] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.756850 1677 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:55.757804 1677 ts_tablet_manager.cc:1367] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20240501 14:00:55.761503 1690 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.762652 1690 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:55.768309 1673 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:55.768568 1673 ts_tablet_manager.cc:1367] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.006s
I20240501 14:00:55.769881 1673 raft_consensus.cc:348] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.769532 1677 raft_consensus.cc:348] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.770155 1677 raft_consensus.cc:374] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.770249 1673 raft_consensus.cc:374] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.770295 1677 raft_consensus.cc:729] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.770422 1673 raft_consensus.cc:729] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.770707 1673 consensus_queue.cc:260] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.770646 1677 consensus_queue.cc:260] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.772641 1677 ts_tablet_manager.cc:1398] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:00:55.773427 1677 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:55.774235 1673 ts_tablet_manager.cc:1398] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:55.780457 1677 tablet_bootstrap.cc:654] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.784065 29099 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]). Expected version 1 got 0
I20240501 14:00:55.784482 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad] alter complete (version 1)
I20240501 14:00:55.808440 1677 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:55.809453 1649 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:55.809720 1649 ts_tablet_manager.cc:1367] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.021s
I20240501 14:00:55.811453 1698 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.812556 29099 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]). Expected version 1 got 0
I20240501 14:00:55.814817 1698 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:55.816840 1649 raft_consensus.cc:348] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.817267 1677 ts_tablet_manager.cc:1367] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.002s
I20240501 14:00:55.818074 1677 raft_consensus.cc:348] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.818336 1649 raft_consensus.cc:374] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.818517 1677 raft_consensus.cc:374] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.818884 1649 raft_consensus.cc:729] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.819033 1677 raft_consensus.cc:729] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.819921 1677 consensus_queue.cc:260] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.820169 1649 consensus_queue.cc:260] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.820472 1655 raft_consensus.cc:482] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.820588 1655 raft_consensus.cc:504] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.821535 1655 leader_election.cc:288] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:55.822808 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
W20240501 14:00:55.823338 32032 leader_election.cc:341] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:55.823737 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:55.823979 31965 raft_consensus.cc:2454] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:55.824426 32032 leader_election.cc:302] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:00:55.825253 1655 raft_consensus.cc:2781] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.825361 1655 raft_consensus.cc:482] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.825444 1655 raft_consensus.cc:3037] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.828053 1655 raft_consensus.cc:504] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.829200 1672 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:55.829362 1672 ts_tablet_manager.cc:1367] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.007s
I20240501 14:00:55.829286 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
W20240501 14:00:55.830204 32032 leader_election.cc:341] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:55.830121 1672 raft_consensus.cc:348] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.830919 1672 raft_consensus.cc:374] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.830956 1655 leader_election.cc:288] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:55.831089 1672 raft_consensus.cc:729] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.831146 1439 raft_consensus.cc:482] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.831321 1439 raft_consensus.cc:504] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.831395 1672 consensus_queue.cc:260] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.832012 1439 leader_election.cc:288] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:55.832553 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7a7501266946dca6bc838d10524c4a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:55.832767 32227 raft_consensus.cc:2454] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:55.833199 32295 leader_election.cc:302] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:55.833695 1672 ts_tablet_manager.cc:1398] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:55.833813 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:55.834059 1672 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:55.834061 31965 raft_consensus.cc:3037] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.834292 1439 raft_consensus.cc:2781] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.834409 1439 raft_consensus.cc:482] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.834507 1439 raft_consensus.cc:3037] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.835538 1672 tablet_bootstrap.cc:654] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.833912 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7a7501266946dca6bc838d10524c4a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:55.837165 31963 raft_consensus.cc:2454] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:55.838011 1439 raft_consensus.cc:504] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.839090 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7a7501266946dca6bc838d10524c4a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:55.839325 32227 raft_consensus.cc:3037] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.842097 32227 raft_consensus.cc:2454] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:55.842603 32295 leader_election.cc:302] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:55.844434 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7a7501266946dca6bc838d10524c4a" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:55.844640 31963 raft_consensus.cc:3037] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.846112 31965 raft_consensus.cc:2454] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:55.847530 31963 raft_consensus.cc:2454] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:55.848367 32032 leader_election.cc:302] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:00:55.849025 1650 raft_consensus.cc:2781] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.849197 1650 raft_consensus.cc:686] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:55.849517 1650 consensus_queue.cc:237] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:55.851352 1439 leader_election.cc:288] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:55.851918 1439 raft_consensus.cc:2781] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.852504 1439 raft_consensus.cc:686] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:55.853680 1439 consensus_queue.cc:237] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:55.866776 1672 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:55.867548 1672 ts_tablet_manager.cc:1367] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.004s
I20240501 14:00:55.869135 1672 raft_consensus.cc:348] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.870038 1672 raft_consensus.cc:374] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.870293 1672 raft_consensus.cc:729] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.872303 1672 consensus_queue.cc:260] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.873520 1672 ts_tablet_manager.cc:1398] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:55.888558 29099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:55.898986 1690 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.899613 1690 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:00:55.902287 1698 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.902856 1698 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:00:55.908990 1692 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.909505 1692 tablet.cc:1620] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:55.917701 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d4c80bd0c14b4c9f9baba73e55c8a2ad] alter complete (version 2)
W20240501 14:00:55.936863 32295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 94285.1us
I20240501 14:00:55.938445 1527 raft_consensus.cc:482] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.938645 1527 raft_consensus.cc:504] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.939316 1527 leader_election.cc:288] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:55.940436 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f3644410a24fc1a9457ff46a10c180" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:55.940706 32097 raft_consensus.cc:2454] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:55.941224 29312 leader_election.cc:302] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:55.942531 1527 raft_consensus.cc:2781] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.942644 1527 raft_consensus.cc:482] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.942714 1527 raft_consensus.cc:3037] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.944625 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f3644410a24fc1a9457ff46a10c180" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:55.944844 32360 raft_consensus.cc:2454] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:55.945878 1527 raft_consensus.cc:504] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.945811 29099 catalog_manager.cc:5484] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:55.947036 1527 leader_election.cc:288] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:55.947324 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f3644410a24fc1a9457ff46a10c180" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:55.947582 32097 raft_consensus.cc:3037] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.951208 32097 raft_consensus.cc:2454] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:55.951383 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f3644410a24fc1a9457ff46a10c180" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:55.951619 32360 raft_consensus.cc:3037] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.951689 29312 leader_election.cc:302] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:55.952016 1527 raft_consensus.cc:2781] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.952163 1527 raft_consensus.cc:686] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:55.952430 1527 consensus_queue.cc:237] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:55.953603 29099 catalog_manager.cc:5484] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:55.954353 32360 raft_consensus.cc:2454] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:55.965493 1677 ts_tablet_manager.cc:1398] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.148s user 0.001s sys 0.002s
I20240501 14:00:55.966428 1649 ts_tablet_manager.cc:1398] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.150s user 0.002s sys 0.000s
I20240501 14:00:55.970736 29097 catalog_manager.cc:5484] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:00:56.005398 29505 consensus_queue.cc:237] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:00:56.007966 32097 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:56.008486 1175 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:56.008605 29637 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:56.010025 1175 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:56.013059 1689 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:56.013934 29637 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:56.020066 29099 catalog_manager.cc:5484] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:56.025293 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:56.026058 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b72e2ac9df4426b071a7e5004e12e1 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51134
I20240501 14:00:56.040234 1716 tablet_replica.cc:330] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:00:56.040722 1716 raft_consensus.cc:2229] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:56.041244 1716 raft_consensus.cc:2258] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:56.042438 1716 ts_tablet_manager.cc:1875] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:56.045420 1690 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:56.047237 1698 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:56.056674 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=d4c80bd0c14b4c9f9baba73e55c8a2ad] alter complete (version 3)
I20240501 14:00:56.058406 1692 tablet.cc:1722] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:56.065755 1716 ts_tablet_manager.cc:1888] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:00:56.066012 1716 log.cc:1196] T 49b72e2ac9df4426b071a7e5004e12e1 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:00:56.069023 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 49b72e2ac9df4426b071a7e5004e12e1 (table rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]) successfully deleted
I20240501 14:00:56.071527 32096 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:00:56.163534 29099 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:56.163805 29099 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:56.168272 29099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 150338a5de7544e29cade37e0ca8f820
I20240501 14:00:56.169255 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 150338a5de7544e29cade37e0ca8f820 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:56 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:56.169401 31945 tablet_service.cc:1514] Processing DeleteTablet for tablet 150338a5de7544e29cade37e0ca8f820 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:00:56.169803 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 150338a5de7544e29cade37e0ca8f820 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:56 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:56.169950 1758 tablet_replica.cc:330] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:00:56.170257 1758 raft_consensus.cc:2229] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:56.170701 1758 raft_consensus.cc:2258] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:56.172013 1758 ts_tablet_manager.cc:1875] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:56.181517 1759 tablet_replica.cc:330] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:56.181856 1759 raft_consensus.cc:2229] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:56.182286 1759 raft_consensus.cc:2258] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:56.182581 1758 ts_tablet_manager.cc:1888] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:56.182770 1758 log.cc:1196] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/150338a5de7544e29cade37e0ca8f820
I20240501 14:00:56.183290 1758 ts_tablet_manager.cc:1909] T 150338a5de7544e29cade37e0ca8f820 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:00:56.184525 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table_new_name [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]) successfully deleted
I20240501 14:00:56.203406 1759 ts_tablet_manager.cc:1875] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:56.219472 1757 tablet_replica.cc:330] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:56.220449 1757 raft_consensus.cc:2229] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:56.220755 1757 raft_consensus.cc:2258] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:56.229203 1757 ts_tablet_manager.cc:1875] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:56.230396 1759 ts_tablet_manager.cc:1888] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:56.230743 1759 log.cc:1196] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/150338a5de7544e29cade37e0ca8f820
I20240501 14:00:56.231490 1759 ts_tablet_manager.cc:1909] T 150338a5de7544e29cade37e0ca8f820 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:00:56.234189 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table_new_name [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]) successfully deleted
I20240501 14:00:56.237648 1650 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.237882 1757 ts_tablet_manager.cc:1888] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:56.238013 1757 log.cc:1196] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/150338a5de7544e29cade37e0ca8f820
I20240501 14:00:56.238812 1757 ts_tablet_manager.cc:1909] T 150338a5de7544e29cade37e0ca8f820 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:56.239876 1697 raft_consensus.cc:482] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.240036 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 150338a5de7544e29cade37e0ca8f820 (table rebalancer_dynamic_table_new_name [id=d4c80bd0c14b4c9f9baba73e55c8a2ad]) successfully deleted
I20240501 14:00:56.240034 1697 raft_consensus.cc:504] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.240892 1697 leader_election.cc:288] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:56.241632 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:56.242729 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb574dc421745618ba2a11f0b3f9110" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:56.243400 29312 leader_election.cc:302] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87d188552494b4590afc37a85b98c35; no voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:56.244086 1697 raft_consensus.cc:2726] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:56.258448 29376 raft_consensus.cc:3037] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.263079 1650 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.271950 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:56.285833 32340 tablet_service.cc:1467] Processing CreateTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]), partition=
I20240501 14:00:56.286257 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac843f36e84cb99f7d92a33e3c7d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.286613 32206 tablet_service.cc:1467] Processing CreateTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]), partition=
I20240501 14:00:56.286967 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac843f36e84cb99f7d92a33e3c7d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.287029 29487 tablet_service.cc:1467] Processing CreateTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]), partition=
I20240501 14:00:56.287549 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac843f36e84cb99f7d92a33e3c7d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.290654 1702 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.295722 1677 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:56.297358 1677 tablet_bootstrap.cc:654] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.314895 1702 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.316565 1781 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:56.319459 1781 tablet_bootstrap.cc:654] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.329569 1781 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:56.330578 1781 ts_tablet_manager.cc:1367] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240501 14:00:56.331889 1781 raft_consensus.cc:348] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.333513 1781 raft_consensus.cc:374] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.333860 1781 raft_consensus.cc:729] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.334527 1781 consensus_queue.cc:260] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.335929 1781 ts_tablet_manager.cc:1398] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:56.361513 1697 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.385206 29226 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:56.392024 1697 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.395514 1677 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:56.395820 1677 ts_tablet_manager.cc:1367] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.100s user 0.030s sys 0.041s
I20240501 14:00:56.396935 1677 raft_consensus.cc:348] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.397279 1677 raft_consensus.cc:374] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.397459 1677 raft_consensus.cc:729] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.397794 1777 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:56.397855 1677 consensus_queue.cc:260] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.399030 1677 ts_tablet_manager.cc:1398] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:56.440488 32077 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:56.462687 1777 tablet_bootstrap.cc:654] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.536864 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:56.538616 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:56.545653 1777 tablet_bootstrap.cc:492] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:56.545897 1777 ts_tablet_manager.cc:1367] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.148s user 0.004s sys 0.009s
I20240501 14:00:56.549064 1777 raft_consensus.cc:348] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.549486 1777 raft_consensus.cc:374] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.549646 1777 raft_consensus.cc:729] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.550006 1777 consensus_queue.cc:260] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.554200 1777 ts_tablet_manager.cc:1398] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:56.560034 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 2926d8d685254b0ca2948c56757a192b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.560544 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2926d8d685254b0ca2948c56757a192b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.561062 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 2926d8d685254b0ca2948c56757a192b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.561681 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2926d8d685254b0ca2948c56757a192b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.570783 29226 tablet_service.cc:1467] Processing CreateTablet for tablet e67dc79a086048a8839ae386857652ea (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.571370 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67dc79a086048a8839ae386857652ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.573944 29487 tablet_service.cc:1467] Processing CreateTablet for tablet e67dc79a086048a8839ae386857652ea (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.574544 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67dc79a086048a8839ae386857652ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.574887 1797 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:56.576525 1797 tablet_bootstrap.cc:654] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.577569 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 2926d8d685254b0ca2948c56757a192b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.578094 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2926d8d685254b0ca2948c56757a192b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.583328 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 7fce4b1a15bc48e9b8042869d5c18d16 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.583874 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fce4b1a15bc48e9b8042869d5c18d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.592051 1677 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:56.582665 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 7fce4b1a15bc48e9b8042869d5c18d16 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.592849 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fce4b1a15bc48e9b8042869d5c18d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.593824 1677 tablet_bootstrap.cc:654] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.595361 1777 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:56.597039 1777 tablet_bootstrap.cc:654] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.607017 1781 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:56.608897 1781 tablet_bootstrap.cc:654] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.611611 32340 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:56.617646 1777 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:56.617971 1777 ts_tablet_manager.cc:1367] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20240501 14:00:56.620801 1705 raft_consensus.cc:482] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.621011 1705 raft_consensus.cc:504] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.621795 1705 leader_election.cc:288] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:56.622014 32206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:56.622705 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac843f36e84cb99f7d92a33e3c7d91" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:56.622943 29507 raft_consensus.cc:2454] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:56.623401 32295 leader_election.cc:302] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:56.623865 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac843f36e84cb99f7d92a33e3c7d91" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:56.624109 32227 raft_consensus.cc:2454] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:56.624925 1705 raft_consensus.cc:2781] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.625123 1705 raft_consensus.cc:482] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.625221 1705 raft_consensus.cc:3037] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.620527 1777 raft_consensus.cc:348] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.627209 1777 raft_consensus.cc:374] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.631450 1777 raft_consensus.cc:729] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.631870 1777 consensus_queue.cc:260] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.637401 1705 raft_consensus.cc:504] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.638566 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac843f36e84cb99f7d92a33e3c7d91" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:56.638780 29507 raft_consensus.cc:3037] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.642041 29507 raft_consensus.cc:2454] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:56.642617 32295 leader_election.cc:302] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:56.643431 1702 raft_consensus.cc:2781] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.643592 1702 raft_consensus.cc:686] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:56.643728 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac843f36e84cb99f7d92a33e3c7d91" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:56.644004 32228 raft_consensus.cc:3037] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.643922 1702 consensus_queue.cc:237] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.645121 1705 leader_election.cc:288] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:56.645606 1777 ts_tablet_manager.cc:1398] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240501 14:00:56.647519 32228 raft_consensus.cc:2454] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:56.655972 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 7fce4b1a15bc48e9b8042869d5c18d16 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.656562 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fce4b1a15bc48e9b8042869d5c18d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.667606 32206 tablet_service.cc:1467] Processing CreateTablet for tablet e67dc79a086048a8839ae386857652ea (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.669834 1797 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:56.670063 1797 ts_tablet_manager.cc:1367] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.000s
I20240501 14:00:56.668243 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67dc79a086048a8839ae386857652ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.670861 1797 raft_consensus.cc:348] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.671236 1797 raft_consensus.cc:374] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.671367 1797 raft_consensus.cc:729] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.671666 1797 consensus_queue.cc:260] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.673627 1797 ts_tablet_manager.cc:1398] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:56.673928 1797 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:56.675611 1797 tablet_bootstrap.cc:654] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.683838 1797 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:56.684082 1803 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:56.684243 1797 ts_tablet_manager.cc:1367] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:00:56.689222 29097 catalog_manager.cc:5484] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:56.692723 1677 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:56.692919 1677 ts_tablet_manager.cc:1367] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.010s
I20240501 14:00:56.693796 1677 raft_consensus.cc:348] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.697559 1803 tablet_bootstrap.cc:654] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.698565 1677 raft_consensus.cc:374] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.699684 1677 raft_consensus.cc:729] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.700110 1677 consensus_queue.cc:260] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.702613 1781 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:56.702903 1781 ts_tablet_manager.cc:1367] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.008s
I20240501 14:00:56.704171 1781 raft_consensus.cc:348] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.704560 1781 raft_consensus.cc:374] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.704689 1781 raft_consensus.cc:729] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.705199 1781 consensus_queue.cc:260] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.707319 1677 ts_tablet_manager.cc:1398] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:00:56.714968 1781 ts_tablet_manager.cc:1398] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:00:56.720718 1799 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:56.726032 1799 tablet_bootstrap.cc:654] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.730751 1781 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:56.685209 1797 raft_consensus.cc:348] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.733703 1797 raft_consensus.cc:374] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.734221 1797 raft_consensus.cc:729] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.735011 1797 consensus_queue.cc:260] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.740437 1781 tablet_bootstrap.cc:654] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.741043 1797 ts_tablet_manager.cc:1398] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20240501 14:00:56.750473 1781 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:56.753716 1781 ts_tablet_manager.cc:1367] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:00:56.754799 1799 tablet_bootstrap.cc:492] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:56.755052 1799 ts_tablet_manager.cc:1367] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240501 14:00:56.755960 1799 raft_consensus.cc:348] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.756274 1799 raft_consensus.cc:374] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.756398 1799 raft_consensus.cc:729] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.756726 1799 consensus_queue.cc:260] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.758601 1175 raft_consensus.cc:482] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.758774 1175 raft_consensus.cc:504] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.759521 1175 leader_election.cc:288] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:00:56.760800 1803 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:56.761180 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:56.761209 1803 ts_tablet_manager.cc:1367] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.002s
W20240501 14:00:56.761962 29440 leader_election.cc:341] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:56.762328 1799 ts_tablet_manager.cc:1398] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:00:56.762857 1803 raft_consensus.cc:348] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.763163 1803 raft_consensus.cc:374] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.763307 1803 raft_consensus.cc:729] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.763648 1803 consensus_queue.cc:260] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.768337 1803 ts_tablet_manager.cc:1398] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:56.760807 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
W20240501 14:00:56.773772 29441 leader_election.cc:341] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:56.773934 29441 leader_election.cc:302] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:56.774406 1175 raft_consensus.cc:2726] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:56.774570 1781 raft_consensus.cc:348] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.774827 1781 raft_consensus.cc:374] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.774927 1781 raft_consensus.cc:729] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.775184 1781 consensus_queue.cc:260] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.789212 1781 ts_tablet_manager.cc:1398] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240501 14:00:56.790604 1781 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:56.790936 29356 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:56.792310 1781 tablet_bootstrap.cc:654] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.797056 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:56.802472 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:56.806411 32227 raft_consensus.cc:1261] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:56.806636 29507 raft_consensus.cc:1261] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:56.807521 1705 consensus_queue.cc:1035] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.807991 1705 consensus_queue.cc:1035] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.852100 31945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:56.857224 1650 raft_consensus.cc:482] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.857394 1650 raft_consensus.cc:504] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.857862 29097 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]). Expected version 1 got 0
I20240501 14:00:56.858096 1650 leader_election.cc:288] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:56.858765 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2926d8d685254b0ca2948c56757a192b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:56.859054 31963 raft_consensus.cc:2454] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:56.859158 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2926d8d685254b0ca2948c56757a192b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:56.859472 32226 raft_consensus.cc:2454] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:56.860047 32032 leader_election.cc:302] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:56.860361 1650 raft_consensus.cc:2781] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.860513 1650 raft_consensus.cc:482] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.860606 1650 raft_consensus.cc:3037] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.864236 1650 raft_consensus.cc:504] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.865094 1650 leader_election.cc:288] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:56.869100 1781 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:56.869314 1781 ts_tablet_manager.cc:1367] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20240501 14:00:56.870082 1781 raft_consensus.cc:348] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.870213 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2926d8d685254b0ca2948c56757a192b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:56.870302 1781 raft_consensus.cc:374] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.870379 1781 raft_consensus.cc:729] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.870438 31963 raft_consensus.cc:3037] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.874400 1821 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.875062 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2926d8d685254b0ca2948c56757a192b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:56.875285 32227 raft_consensus.cc:3037] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.876163 1650 raft_consensus.cc:482] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.876317 1650 raft_consensus.cc:504] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.876807 1821 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:00:56.877635 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fce4b1a15bc48e9b8042869d5c18d16" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:56.877871 32360 raft_consensus.cc:2454] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:56.878404 32031 leader_election.cc:302] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:56.878648 31963 raft_consensus.cc:2454] T 2926d8d685254b0ca2948c56757a192b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:56.879086 32032 leader_election.cc:302] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:56.879444 1650 leader_election.cc:288] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:56.879621 1650 raft_consensus.cc:2781] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.879719 1650 raft_consensus.cc:482] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.879806 1650 raft_consensus.cc:3037] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.880731 29105 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]). Expected version 1 got 0
I20240501 14:00:56.884078 1650 raft_consensus.cc:504] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.884754 1650 leader_election.cc:288] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:56.886962 1818 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.887817 1818 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:56.889482 1781 consensus_queue.cc:260] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:56.890839 1823 raft_consensus.cc:2781] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.890915 32227 raft_consensus.cc:2454] T 2926d8d685254b0ca2948c56757a192b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:56.891628 1823 raft_consensus.cc:686] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:56.891779 1781 ts_tablet_manager.cc:1398] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:00:56.895535 1823 consensus_queue.cc:237] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:56.899708 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3] alter complete (version 1)
I20240501 14:00:56.900112 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fce4b1a15bc48e9b8042869d5c18d16" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:56.902323 32227 raft_consensus.cc:2454] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:56.902614 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fce4b1a15bc48e9b8042869d5c18d16" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:56.902840 32360 raft_consensus.cc:3037] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.903965 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fce4b1a15bc48e9b8042869d5c18d16" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:56.904158 32226 raft_consensus.cc:3037] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.907301 32226 raft_consensus.cc:2454] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:56.908073 32031 leader_election.cc:302] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:56.908577 1823 raft_consensus.cc:2781] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.908834 1823 raft_consensus.cc:686] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:56.908866 32360 raft_consensus.cc:2454] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:56.911397 29096 catalog_manager.cc:5484] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:56.911761 1823 consensus_queue.cc:237] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:56.925221 29096 catalog_manager.cc:5484] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:56.941774 1819 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.942689 1819 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:56.954895 29096 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]). Expected version 1 got 0
I20240501 14:00:57.014827 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:57.023878 1660 raft_consensus.cc:482] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.024128 1660 raft_consensus.cc:504] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:57.024878 1660 leader_election.cc:288] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:57.026211 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:57.026440 32226 raft_consensus.cc:2454] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:57.027747 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:57.027962 29507 raft_consensus.cc:2454] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:57.028497 29180 leader_election.cc:302] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:57.028751 1448 raft_consensus.cc:2781] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.028874 1448 raft_consensus.cc:482] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.028954 1448 raft_consensus.cc:3037] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.029908 1818 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.030462 1818 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:57.031785 1448 raft_consensus.cc:504] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:57.032371 1448 leader_election.cc:288] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:00:57.032500 1821 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.033021 1821 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:00:57.035001 1819 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.035491 1819 tablet.cc:1620] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:57.036712 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:57.036901 32228 raft_consensus.cc:3037] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.040773 32228 raft_consensus.cc:2454] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:57.043205 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67dc79a086048a8839ae386857652ea" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:57.043447 29505 raft_consensus.cc:3037] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.045451 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=cb62ba6a1d684c3ebb8ed0c0602be3d3] alter complete (version 2)
I20240501 14:00:57.046216 29505 raft_consensus.cc:2454] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:57.047106 29180 leader_election.cc:302] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:00:57.047931 1448 raft_consensus.cc:2781] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.048087 1448 raft_consensus.cc:686] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:57.048372 1448 consensus_queue.cc:237] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:00:57.055133 29105 catalog_manager.cc:5484] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:00:57.099254 29244 consensus_queue.cc:237] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.101706 29635 raft_consensus.cc:1261] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:57.102035 31963 raft_consensus.cc:1261] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:57.102627 1660 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:57.103355 1660 consensus_queue.cc:1035] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:57.105445 1448 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:57.106982 29637 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:57.110908 31963 raft_consensus.cc:2932] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:57.118636 29105 catalog_manager.cc:5484] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:57.124444 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 60527f81f34b45c199bf52bf0df53ee2 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:57.138371 1847 tablet_replica.cc:330] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:57.138643 1847 raft_consensus.cc:2229] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.138960 1847 raft_consensus.cc:2258] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.139717 1847 ts_tablet_manager.cc:1875] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:57.147204 1847 ts_tablet_manager.cc:1888] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:00:57.147323 1847 log.cc:1196] T 60527f81f34b45c199bf52bf0df53ee2 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/60527f81f34b45c199bf52bf0df53ee2
I20240501 14:00:57.148051 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 60527f81f34b45c199bf52bf0df53ee2 (table rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]) successfully deleted
I20240501 14:00:57.151845 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:57.160140 1818 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.165256 1821 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.170660 1819 tablet.cc:1722] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.172811 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=cb62ba6a1d684c3ebb8ed0c0602be3d3] alter complete (version 3)
I20240501 14:00:57.282415 29099 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:57.282706 29099 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:57.287842 29099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet caac843f36e84cb99f7d92a33e3c7d91
I20240501 14:00:57.288440 1650 consensus_queue.cc:1035] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.288695 32340 tablet_service.cc:1514] Processing DeleteTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:57.289825 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:57.290789 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet caac843f36e84cb99f7d92a33e3c7d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:57.291116 1847 tablet_replica.cc:330] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:57.291376 1847 raft_consensus.cc:2229] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.291595 1847 raft_consensus.cc:2258] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.292840 1847 ts_tablet_manager.cc:1875] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.300021 1885 tablet_replica.cc:330] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:57.300432 1885 raft_consensus.cc:2229] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:57.301620 1885 raft_consensus.cc:2258] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.303032 1885 ts_tablet_manager.cc:1875] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.303725 1887 tablet_replica.cc:330] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:57.304165 1887 raft_consensus.cc:2229] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.304736 1887 raft_consensus.cc:2258] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.306167 1887 ts_tablet_manager.cc:1875] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.309542 1847 ts_tablet_manager.cc:1888] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.309773 1847 log.cc:1196] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/caac843f36e84cb99f7d92a33e3c7d91
I20240501 14:00:57.311902 1847 ts_tablet_manager.cc:1909] T caac843f36e84cb99f7d92a33e3c7d91 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:00:57.315773 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table_new_name [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]) successfully deleted
I20240501 14:00:57.320809 1885 ts_tablet_manager.cc:1888] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.321038 1885 log.cc:1196] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/caac843f36e84cb99f7d92a33e3c7d91
I20240501 14:00:57.321578 1885 ts_tablet_manager.cc:1909] T caac843f36e84cb99f7d92a33e3c7d91 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:57.323033 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table_new_name [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]) successfully deleted
I20240501 14:00:57.327128 1650 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.328095 1887 ts_tablet_manager.cc:1888] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.328311 1887 log.cc:1196] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/caac843f36e84cb99f7d92a33e3c7d91
I20240501 14:00:57.329069 1887 ts_tablet_manager.cc:1909] T caac843f36e84cb99f7d92a33e3c7d91 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:57.331848 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet caac843f36e84cb99f7d92a33e3c7d91 (table rebalancer_dynamic_table_new_name [id=cb62ba6a1d684c3ebb8ed0c0602be3d3]) successfully deleted
I20240501 14:00:57.345877 1650 consensus_queue.cc:1035] T 2926d8d685254b0ca2948c56757a192b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.354032 1650 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.395313 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:57.402846 32077 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:57.423440 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 6a75b2a4a43140069d98952220d023ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]), partition=
I20240501 14:00:57.423882 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a75b2a4a43140069d98952220d023ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.425742 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 6a75b2a4a43140069d98952220d023ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]), partition=
I20240501 14:00:57.426170 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a75b2a4a43140069d98952220d023ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.427145 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 6a75b2a4a43140069d98952220d023ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]), partition=
I20240501 14:00:57.427546 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a75b2a4a43140069d98952220d023ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.443095 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:57.460034 1912 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:57.464192 1912 tablet_bootstrap.cc:654] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.486433 1915 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:57.490527 1915 tablet_bootstrap.cc:654] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.493907 32342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:57.498607 1911 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:57.510418 31945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:57.519155 1448 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.519517 1911 tablet_bootstrap.cc:654] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.530510 1912 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:57.530752 1912 ts_tablet_manager.cc:1367] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.008s
I20240501 14:00:57.531697 1912 raft_consensus.cc:348] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.532063 1912 raft_consensus.cc:374] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.532213 1912 raft_consensus.cc:729] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.532547 1912 consensus_queue.cc:260] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.536435 1912 ts_tablet_manager.cc:1398] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:00:57.546321 29226 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:00:57.553108 32206 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:57.558121 1911 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:57.558424 1911 ts_tablet_manager.cc:1367] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.002s
I20240501 14:00:57.559264 1911 raft_consensus.cc:348] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.559664 1911 raft_consensus.cc:374] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.559794 1911 raft_consensus.cc:729] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.560089 1911 consensus_queue.cc:260] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.563306 1911 ts_tablet_manager.cc:1398] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:57.581583 1448 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.587987 29356 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:57.589516 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:57.610245 1915 tablet_bootstrap.cc:492] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:57.610533 1915 ts_tablet_manager.cc:1367] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.012s
I20240501 14:00:57.611085 31945 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6e3129a2f4bf89a36cdca5eaeb376 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.611580 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6e3129a2f4bf89a36cdca5eaeb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.611768 1915 raft_consensus.cc:348] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.611976 1915 raft_consensus.cc:374] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.612049 1915 raft_consensus.cc:729] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.614079 1915 consensus_queue.cc:260] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.615798 1915 ts_tablet_manager.cc:1398] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:57.619000 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6e3129a2f4bf89a36cdca5eaeb376 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.619521 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6e3129a2f4bf89a36cdca5eaeb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.622274 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6e3129a2f4bf89a36cdca5eaeb376 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.622776 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6e3129a2f4bf89a36cdca5eaeb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.622824 32342 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed5a5797424f0f88c2b3c711468e6b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.623154 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed5a5797424f0f88c2b3c711468e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.624783 29226 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed5a5797424f0f88c2b3c711468e6b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.625231 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed5a5797424f0f88c2b3c711468e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.629436 1911 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:57.630853 31944 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed5a5797424f0f88c2b3c711468e6b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.631073 1911 tablet_bootstrap.cc:654] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.631294 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed5a5797424f0f88c2b3c711468e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.644263 1911 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:57.644951 1911 ts_tablet_manager.cc:1367] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20240501 14:00:57.646816 29487 tablet_service.cc:1467] Processing CreateTablet for tablet c3f0609897bd4890952f7535cfe737f2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.647361 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f0609897bd4890952f7535cfe737f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.646389 1911 raft_consensus.cc:348] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.648635 1911 raft_consensus.cc:374] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.648710 1911 raft_consensus.cc:729] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.648960 1911 consensus_queue.cc:260] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.651710 1911 ts_tablet_manager.cc:1398] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:00:57.652006 1911 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:57.657567 1925 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:57.663177 1925 tablet_bootstrap.cc:654] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.657044 1911 tablet_bootstrap.cc:654] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.670867 32077 tablet_service.cc:1467] Processing CreateTablet for tablet c3f0609897bd4890952f7535cfe737f2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.671516 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f0609897bd4890952f7535cfe737f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.672726 1911 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:57.672930 1911 ts_tablet_manager.cc:1367] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.002s
I20240501 14:00:57.673739 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:57.674291 1911 raft_consensus.cc:348] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.674587 1911 raft_consensus.cc:374] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.674744 1911 raft_consensus.cc:729] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.675086 1911 consensus_queue.cc:260] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.675791 1925 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:57.675997 1925 ts_tablet_manager.cc:1367] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20240501 14:00:57.676743 1911 ts_tablet_manager.cc:1398] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:57.676862 1925 raft_consensus.cc:348] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.677116 1925 raft_consensus.cc:374] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.677222 1925 raft_consensus.cc:729] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.677492 1925 consensus_queue.cc:260] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.678153 1925 ts_tablet_manager.cc:1398] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:57.687067 31943 tablet_service.cc:1467] Processing CreateTablet for tablet c3f0609897bd4890952f7535cfe737f2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.687568 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f0609897bd4890952f7535cfe737f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.691071 1705 raft_consensus.cc:482] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.691239 1705 raft_consensus.cc:504] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.691957 1705 leader_election.cc:288] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
W20240501 14:00:57.693970 32296 leader_election.cc:341] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Not found: Tablet not found: e5ed5a5797424f0f88c2b3c711468e6b
I20240501 14:00:57.696648 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed5a5797424f0f88c2b3c711468e6b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
W20240501 14:00:57.697268 32294 leader_election.cc:341] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:57.697397 32294 leader_election.cc:302] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:57.697818 1705 raft_consensus.cc:2726] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:57.709179 1930 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:57.710500 1929 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:57.716646 1929 tablet_bootstrap.cc:654] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.717404 1930 tablet_bootstrap.cc:654] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.725879 1928 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:57.729388 1929 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:57.730377 1930 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:57.730624 1930 ts_tablet_manager.cc:1367] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 14:00:57.731642 1930 raft_consensus.cc:348] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.731927 1930 raft_consensus.cc:374] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.732065 1930 raft_consensus.cc:729] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.732322 1930 consensus_queue.cc:260] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.732658 1929 ts_tablet_manager.cc:1367] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240501 14:00:57.733474 1930 ts_tablet_manager.cc:1398] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:57.734019 1929 raft_consensus.cc:348] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.734184 1929 raft_consensus.cc:374] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.734294 1929 raft_consensus.cc:729] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.734927 1929 consensus_queue.cc:260] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.737109 1929 ts_tablet_manager.cc:1398] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:57.737861 1929 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:57.739692 1929 tablet_bootstrap.cc:654] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.743044 1912 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:57.745002 1912 tablet_bootstrap.cc:654] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.754561 1928 tablet_bootstrap.cc:654] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.756922 1912 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:57.757138 1912 ts_tablet_manager.cc:1367] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240501 14:00:57.757570 1448 raft_consensus.cc:482] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.757737 1448 raft_consensus.cc:504] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.757953 1912 raft_consensus.cc:348] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.758173 1912 raft_consensus.cc:374] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.758262 1912 raft_consensus.cc:729] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.758481 1448 leader_election.cc:288] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:57.758517 1912 consensus_queue.cc:260] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.759697 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed5a5797424f0f88c2b3c711468e6b" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:57.759938 31963 raft_consensus.cc:2454] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:57.760169 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed5a5797424f0f88c2b3c711468e6b" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:57.760413 32362 raft_consensus.cc:2454] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:57.760838 29181 leader_election.cc:302] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:57.761612 1660 raft_consensus.cc:2781] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.761731 1660 raft_consensus.cc:482] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.761832 1660 raft_consensus.cc:3037] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.764637 1660 raft_consensus.cc:504] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.768368 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed5a5797424f0f88c2b3c711468e6b" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:57.768621 32362 raft_consensus.cc:3037] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.768966 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed5a5797424f0f88c2b3c711468e6b" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:57.769212 31963 raft_consensus.cc:3037] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.771400 32362 raft_consensus.cc:2454] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:57.771972 31963 raft_consensus.cc:2454] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:57.772508 29180 leader_election.cc:302] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:57.773176 1448 raft_consensus.cc:2781] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.776156 1912 ts_tablet_manager.cc:1398] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:00:57.777022 1660 leader_election.cc:288] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:00:57.787103 1929 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:57.787312 1929 ts_tablet_manager.cc:1367] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.000s
I20240501 14:00:57.789130 1929 raft_consensus.cc:348] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.789386 1929 raft_consensus.cc:374] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.789435 949 raft_consensus.cc:482] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.789494 1929 raft_consensus.cc:729] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.789625 949 raft_consensus.cc:504] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.790313 949 leader_election.cc:288] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:57.790691 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6e3129a2f4bf89a36cdca5eaeb376" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:57.790841 1929 consensus_queue.cc:260] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.791132 31963 raft_consensus.cc:2454] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:57.791635 32163 leader_election.cc:302] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:57.791970 1795 raft_consensus.cc:2781] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.792061 1795 raft_consensus.cc:482] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.792146 1795 raft_consensus.cc:3037] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.794905 1795 raft_consensus.cc:504] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.798952 1795 leader_election.cc:288] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:57.799402 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6e3129a2f4bf89a36cdca5eaeb376" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:57.799649 32360 raft_consensus.cc:3037] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.801621 1929 ts_tablet_manager.cc:1398] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:00:57.802589 32360 raft_consensus.cc:2454] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:57.803128 32162 leader_election.cc:302] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:57.803232 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6e3129a2f4bf89a36cdca5eaeb376" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:57.803453 949 raft_consensus.cc:2781] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.803503 32362 raft_consensus.cc:2362] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1. Re-sending same reply.
I20240501 14:00:57.803599 949 raft_consensus.cc:686] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:57.803900 949 consensus_queue.cc:237] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:57.804498 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6e3129a2f4bf89a36cdca5eaeb376" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:57.804800 31963 raft_consensus.cc:3037] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.808357 1928 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:57.808549 1928 ts_tablet_manager.cc:1367] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20240501 14:00:57.808617 1929 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:57.809485 1928 raft_consensus.cc:348] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.809702 1928 raft_consensus.cc:374] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.809790 1928 raft_consensus.cc:729] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.810083 1928 consensus_queue.cc:260] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.811794 1928 ts_tablet_manager.cc:1398] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:57.812152 31963 raft_consensus.cc:2454] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:57.813892 1929 tablet_bootstrap.cc:654] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.826997 1929 tablet_bootstrap.cc:492] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:57.827204 1929 ts_tablet_manager.cc:1367] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240501 14:00:57.828183 1929 raft_consensus.cc:348] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.828451 1929 raft_consensus.cc:374] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.828557 1929 raft_consensus.cc:729] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.828831 1929 consensus_queue.cc:260] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.835970 1929 ts_tablet_manager.cc:1398] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:00:57.848503 1448 raft_consensus.cc:686] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:57.850428 1448 consensus_queue.cc:237] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.854343 29097 catalog_manager.cc:5484] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:57.899155 29097 catalog_manager.cc:5484] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:57.935274 949 raft_consensus.cc:482] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.935467 949 raft_consensus.cc:504] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.936178 949 leader_election.cc:288] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:57.936779 1705 raft_consensus.cc:482] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.936960 1705 raft_consensus.cc:504] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.937830 1705 leader_election.cc:288] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:57.938503 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:57.938709 32360 raft_consensus.cc:2454] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:57.939899 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:57.940146 29635 raft_consensus.cc:2454] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:57.940797 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:57.942286 32294 leader_election.cc:302] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:00:57.942770 32228 raft_consensus.cc:2454] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:00:57.944793 1798 raft_consensus.cc:482] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.945039 1798 raft_consensus.cc:504] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.946064 1798 leader_election.cc:288] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:57.947182 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f0609897bd4890952f7535cfe737f2" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:57.947526 32096 raft_consensus.cc:2454] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:57.949304 32162 leader_election.cc:302] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:57.949362 1944 raft_consensus.cc:2781] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.949502 1944 raft_consensus.cc:482] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.949605 1944 raft_consensus.cc:3037] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.950207 1795 raft_consensus.cc:2781] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.950327 1795 raft_consensus.cc:482] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.950420 1795 raft_consensus.cc:3037] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.955812 31900 leader_election.cc:302] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:57.956279 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f0609897bd4890952f7535cfe737f2" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:57.956521 29507 raft_consensus.cc:2454] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:00:57.956470 1944 raft_consensus.cc:504] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.958038 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:57.958323 29636 raft_consensus.cc:3037] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.958981 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:57.959544 1918 raft_consensus.cc:2781] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.959667 1918 raft_consensus.cc:482] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.959772 1918 raft_consensus.cc:3037] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.963371 1795 raft_consensus.cc:504] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.963591 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:57.963878 32228 raft_consensus.cc:2379] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6edd6a3430d44c38a19c2804e555059f in current term 1: Already voted for candidate 6a446ac802a7457d9f27076781f21d79 in this term.
I20240501 14:00:57.965456 1795 leader_election.cc:288] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:57.965530 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:57.966754 1918 raft_consensus.cc:504] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:57.968165 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f0609897bd4890952f7535cfe737f2" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:57.968463 29507 raft_consensus.cc:3037] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.969843 29636 raft_consensus.cc:2454] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:00:57.974045 32294 leader_election.cc:302] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6a446ac802a7457d9f27076781f21d79
I20240501 14:00:57.974442 1705 raft_consensus.cc:2781] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.974609 1705 raft_consensus.cc:686] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:00:57.974911 1705 consensus_queue.cc:237] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:57.975499 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f0609897bd4890952f7535cfe737f2" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:00:57.975728 32096 raft_consensus.cc:3037] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.981524 1944 leader_election.cc:288] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:57.985728 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a75b2a4a43140069d98952220d023ed" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:57.987825 29104 catalog_manager.cc:5484] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:57.991376 32096 raft_consensus.cc:2454] T c3f0609897bd4890952f7535cfe737f2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:57.992069 31900 leader_election.cc:302] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:00:57.992496 1798 raft_consensus.cc:2781] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.992939 1918 leader_election.cc:288] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:00:57.993340 29507 raft_consensus.cc:2454] T c3f0609897bd4890952f7535cfe737f2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:00:57.996764 32162 leader_election.cc:302] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:57.997121 1795 raft_consensus.cc:2726] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:00:58.003391 1798 raft_consensus.cc:686] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:00:58.003832 1798 consensus_queue.cc:237] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:00:58.032516 29104 catalog_manager.cc:5484] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:00:58.139061 29377 consensus_queue.cc:237] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.140919 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:58.148631 29636 raft_consensus.cc:1261] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:58.149084 32228 raft_consensus.cc:1261] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:58.149363 29507 raft_consensus.cc:1261] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:58.149606 32360 raft_consensus.cc:1261] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:58.150280 1794 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:58.150424 1946 consensus_queue.cc:1035] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.150557 1527 consensus_queue.cc:1035] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:58.150928 1946 consensus_queue.cc:1035] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.180279 1527 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.181891 29097 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]). Expected version 1 got 0
I20240501 14:00:58.186892 29507 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.189253 29097 catalog_manager.cc:5484] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.204206 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ed6fab7d9e46aa8faeb3cf784af0fb with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48910
I20240501 14:00:58.205216 1795 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.216311 1974 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.216948 1974 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:58.220310 32362 raft_consensus.cc:2932] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.221019 1976 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.221645 1976 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:58.227056 1945 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.227528 1977 tablet_replica.cc:330] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:00:58.227886 1977 raft_consensus.cc:2229] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.228051 29097 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]). Expected version 1 got 0
I20240501 14:00:58.228533 1977 raft_consensus.cc:2258] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.229825 1977 ts_tablet_manager.cc:1875] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:58.232712 29104 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4]). Expected version 1 got 0
I20240501 14:00:58.234758 1982 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.235350 1982 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:58.251677 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4] alter complete (version 1)
I20240501 14:00:58.252704 1977 ts_tablet_manager.cc:1888] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:00:58.253196 1977 log.cc:1196] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/67ed6fab7d9e46aa8faeb3cf784af0fb
I20240501 14:00:58.254086 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 67ed6fab7d9e46aa8faeb3cf784af0fb (table rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]) successfully deleted
I20240501 14:00:58.297014 1448 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.315328 1448 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.369825 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:58.377949 1974 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.378525 1974 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:58.378609 1976 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.379101 1976 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:00:58.383116 1982 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.383596 1982 tablet.cc:1620] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:58.391304 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=da182cf2ece4404396e4d453762769f4] alter complete (version 2)
I20240501 14:00:58.497186 1798 consensus_queue.cc:1035] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.503190 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:58.517817 1974 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.519963 1982 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.519963 1976 tablet.cc:1722] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.524200 1948 consensus_queue.cc:1035] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.535713 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=da182cf2ece4404396e4d453762769f4] alter complete (version 3)
I20240501 14:00:58.543135 29100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:58.566361 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 66f7d82d97b6414992f7fdcdaeb83e3d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.566422 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 66f7d82d97b6414992f7fdcdaeb83e3d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.566846 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f7d82d97b6414992f7fdcdaeb83e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.566885 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f7d82d97b6414992f7fdcdaeb83e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.569219 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 054b34b2b424475ca4310b6dd9d97e8e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.569675 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054b34b2b424475ca4310b6dd9d97e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.569875 31944 tablet_service.cc:1467] Processing CreateTablet for tablet db862a636e5a427085a2fa31f6fe8050 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.570318 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db862a636e5a427085a2fa31f6fe8050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.574077 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 054b34b2b424475ca4310b6dd9d97e8e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.574517 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054b34b2b424475ca4310b6dd9d97e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.577351 32206 tablet_service.cc:1467] Processing CreateTablet for tablet db862a636e5a427085a2fa31f6fe8050 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.577749 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db862a636e5a427085a2fa31f6fe8050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.577345 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 054b34b2b424475ca4310b6dd9d97e8e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.581467 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054b34b2b424475ca4310b6dd9d97e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.581638 2040 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:58.582177 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 66f7d82d97b6414992f7fdcdaeb83e3d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.582568 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f7d82d97b6414992f7fdcdaeb83e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.586710 29356 tablet_service.cc:1467] Processing CreateTablet for tablet db862a636e5a427085a2fa31f6fe8050 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.587148 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db862a636e5a427085a2fa31f6fe8050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.605844 32076 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:58.606714 2040 tablet_bootstrap.cc:654] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.613330 2042 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:58.615365 2042 tablet_bootstrap.cc:654] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.617934 2045 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:58.620405 2045 tablet_bootstrap.cc:654] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.620690 29485 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:58.623891 2044 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:58.626158 2049 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:58.626775 2048 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:00:58.627928 2049 tablet_bootstrap.cc:654] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.630496 2044 tablet_bootstrap.cc:654] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.643613 2048 tablet_bootstrap.cc:654] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.647811 2046 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:58.651724 2046 tablet_bootstrap.cc:654] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.653329 29097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.653576 29097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.660449 29097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 6a75b2a4a43140069d98952220d023ed
I20240501 14:00:58.661828 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a75b2a4a43140069d98952220d023ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:00:58.662362 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a75b2a4a43140069d98952220d023ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:58.662590 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a75b2a4a43140069d98952220d023ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:58.669867 2044 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:58.671427 2045 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:58.672473 2040 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:58.672741 2040 ts_tablet_manager.cc:1367] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.000s
I20240501 14:00:58.673671 2040 raft_consensus.cc:348] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.674968 2042 tablet_bootstrap.cc:492] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:58.675112 2061 tablet_replica.cc:330] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:00:58.675246 2045 ts_tablet_manager.cc:1367] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20240501 14:00:58.675666 2044 ts_tablet_manager.cc:1367] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240501 14:00:58.676811 2045 raft_consensus.cc:348] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.677273 2042 ts_tablet_manager.cc:1367] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.000s
I20240501 14:00:58.678429 2040 raft_consensus.cc:374] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.679019 2063 tablet_replica.cc:330] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:58.679212 2040 raft_consensus.cc:729] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.680523 2063 raft_consensus.cc:2229] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.680830 2063 raft_consensus.cc:2258] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.681946 2063 ts_tablet_manager.cc:1875] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.682029 2044 raft_consensus.cc:348] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.683105 2042 raft_consensus.cc:348] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.683373 2040 consensus_queue.cc:260] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.688632 2040 ts_tablet_manager.cc:1398] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:00:58.689245 2045 raft_consensus.cc:374] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.689460 2044 raft_consensus.cc:374] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.689642 2042 raft_consensus.cc:374] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.690074 2061 raft_consensus.cc:2229] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.690336 2061 raft_consensus.cc:2258] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.691957 2048 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:00:58.696527 2046 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:58.710619 2044 raft_consensus.cc:729] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.711092 2044 consensus_queue.cc:260] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.713301 2044 ts_tablet_manager.cc:1398] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 14:00:58.713675 2044 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:58.719107 2046 ts_tablet_manager.cc:1367] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240501 14:00:58.719859 2044 tablet_bootstrap.cc:654] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.720052 2045 raft_consensus.cc:729] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.720866 2046 raft_consensus.cc:348] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.720870 2045 consensus_queue.cc:260] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.722522 2061 ts_tablet_manager.cc:1875] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.723601 2045 ts_tablet_manager.cc:1398] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240501 14:00:58.690277 2062 tablet_replica.cc:330] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:58.724236 2062 raft_consensus.cc:2229] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:58.727941 2062 raft_consensus.cc:2258] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.731753 2062 ts_tablet_manager.cc:1875] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.733052 2046 raft_consensus.cc:374] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.735348 2046 raft_consensus.cc:729] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.735739 2046 consensus_queue.cc:260] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.736871 31944 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:58.743433 2046 ts_tablet_manager.cc:1398] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:00:58.753739 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:58.763142 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:58.775825 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]), partition=
I20240501 14:00:58.776265 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c943f9962fd44e0a3805a9d8e53c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.776674 2042 raft_consensus.cc:729] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.777024 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]), partition=
I20240501 14:00:58.777381 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c943f9962fd44e0a3805a9d8e53c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.784461 29356 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:58.789037 2045 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:58.793184 2048 ts_tablet_manager.cc:1367] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.166s user 0.006s sys 0.000s
I20240501 14:00:58.793805 32342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:58.794601 2048 raft_consensus.cc:348] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.795924 2045 tablet_bootstrap.cc:654] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.799082 2048 raft_consensus.cc:374] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.799342 2048 raft_consensus.cc:729] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.799734 2048 consensus_queue.cc:260] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.800698 2048 ts_tablet_manager.cc:1398] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:58.805184 2042 consensus_queue.cc:260] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.807657 2042 ts_tablet_manager.cc:1398] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.130s user 0.002s sys 0.000s
I20240501 14:00:58.808212 2042 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:58.810791 2042 tablet_bootstrap.cc:654] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.827054 2061 ts_tablet_manager.cc:1888] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.825484 2021 raft_consensus.cc:482] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.827811 2021 raft_consensus.cc:504] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.828547 2021 leader_election.cc:288] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:58.834080 2061 log.cc:1196] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/6a75b2a4a43140069d98952220d023ed
I20240501 14:00:58.834779 2061 ts_tablet_manager.cc:1909] T 6a75b2a4a43140069d98952220d023ed P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:00:58.840603 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table_new_name [id=da182cf2ece4404396e4d453762769f4]) successfully deleted
I20240501 14:00:58.860078 2049 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:58.860249 2049 ts_tablet_manager.cc:1367] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.234s user 0.048s sys 0.049s
I20240501 14:00:58.861058 2049 raft_consensus.cc:348] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.861284 2049 raft_consensus.cc:374] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.861371 2049 raft_consensus.cc:729] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.861653 2049 consensus_queue.cc:260] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.865469 2045 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:58.865698 2045 ts_tablet_manager.cc:1367] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.000s
I20240501 14:00:58.865806 2049 ts_tablet_manager.cc:1398] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:58.866603 2045 raft_consensus.cc:348] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.866919 2045 raft_consensus.cc:374] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.867041 2045 raft_consensus.cc:729] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.867372 2045 consensus_queue.cc:260] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.868122 2045 ts_tablet_manager.cc:1398] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:58.721366 32208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:58.776284 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]), partition=
I20240501 14:00:58.885514 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c943f9962fd44e0a3805a9d8e53c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.830204 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:58.887853 32360 raft_consensus.cc:2454] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:58.888540 32031 leader_election.cc:302] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:58.820595 1945 raft_consensus.cc:482] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.889061 2021 raft_consensus.cc:2781] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.889174 2021 raft_consensus.cc:482] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.889253 2021 raft_consensus.cc:3037] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.889286 1945 raft_consensus.cc:504] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.890429 1945 leader_election.cc:288] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:58.829877 32225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:58.892264 2021 raft_consensus.cc:504] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.892334 32225 raft_consensus.cc:2454] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:58.894685 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:58.894906 32360 raft_consensus.cc:3037] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.896636 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:00:58.897002 32096 raft_consensus.cc:2379] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a446ac802a7457d9f27076781f21d79 in current term 1: Already voted for candidate 0a697f3a1ee544daa7073814ab845d58 in this term.
I20240501 14:00:58.897230 2021 leader_election.cc:288] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:58.898234 32360 raft_consensus.cc:2454] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:58.898689 32031 leader_election.cc:302] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:58.899061 2021 raft_consensus.cc:2781] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.899297 2021 raft_consensus.cc:686] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:58.899574 2021 consensus_queue.cc:237] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.886538 2062 ts_tablet_manager.cc:1888] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.902220 2062 log.cc:1196] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/6a75b2a4a43140069d98952220d023ed
I20240501 14:00:58.906106 2062 ts_tablet_manager.cc:1909] T 6a75b2a4a43140069d98952220d023ed P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:58.907563 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table_new_name [id=da182cf2ece4404396e4d453762769f4]) successfully deleted
I20240501 14:00:58.911621 32225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:58.911877 32225 raft_consensus.cc:3037] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.923681 2042 tablet_bootstrap.cc:492] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:58.923889 2042 ts_tablet_manager.cc:1367] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.000s
I20240501 14:00:58.924710 2042 raft_consensus.cc:348] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.924947 2042 raft_consensus.cc:374] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.925057 2042 raft_consensus.cc:729] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.927379 32225 raft_consensus.cc:2454] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:58.928042 29096 catalog_manager.cc:5484] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.933159 2042 consensus_queue.cc:260] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.933940 2042 ts_tablet_manager.cc:1398] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:00:58.936563 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054b34b2b424475ca4310b6dd9d97e8e" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:58.937201 32360 raft_consensus.cc:2379] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a446ac802a7457d9f27076781f21d79 in current term 1: Already voted for candidate 0a697f3a1ee544daa7073814ab845d58 in this term.
I20240501 14:00:58.937860 2042 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:58.939282 32162 leader_election.cc:302] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:58.939579 1945 raft_consensus.cc:2726] T 054b34b2b424475ca4310b6dd9d97e8e P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:58.941080 2042 tablet_bootstrap.cc:654] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.947855 1939 raft_consensus.cc:482] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.948007 1939 raft_consensus.cc:504] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.948657 1939 leader_election.cc:288] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:58.949575 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f7d82d97b6414992f7fdcdaeb83e3d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:00:58.949813 29507 raft_consensus.cc:2454] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:58.950547 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f7d82d97b6414992f7fdcdaeb83e3d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:58.950798 32360 raft_consensus.cc:2454] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:00:58.951344 29180 leader_election.cc:302] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:58.952261 1939 raft_consensus.cc:2781] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.952347 1939 raft_consensus.cc:482] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.952405 1939 raft_consensus.cc:3037] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.954874 2042 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:58.955018 2042 ts_tablet_manager.cc:1367] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240501 14:00:58.955088 1939 raft_consensus.cc:504] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.955677 1939 leader_election.cc:288] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:00:58.957165 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f7d82d97b6414992f7fdcdaeb83e3d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:00:58.957373 29507 raft_consensus.cc:3037] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.957737 2042 raft_consensus.cc:348] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.957921 2042 raft_consensus.cc:374] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.958014 2042 raft_consensus.cc:729] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.961781 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f7d82d97b6414992f7fdcdaeb83e3d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:58.961997 32360 raft_consensus.cc:3037] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.884150 2063 ts_tablet_manager.cc:1888] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.963661 2063 log.cc:1196] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/6a75b2a4a43140069d98952220d023ed
I20240501 14:00:58.964692 32360 raft_consensus.cc:2454] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:58.963042 2042 consensus_queue.cc:260] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.965165 29180 leader_election.cc:302] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:58.965832 2063 ts_tablet_manager.cc:1909] T 6a75b2a4a43140069d98952220d023ed P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:58.967682 1660 raft_consensus.cc:2781] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.967832 1660 raft_consensus.cc:686] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:00:58.967847 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 6a75b2a4a43140069d98952220d023ed (table rebalancer_dynamic_table_new_name [id=da182cf2ece4404396e4d453762769f4]) successfully deleted
I20240501 14:00:58.968062 1660 consensus_queue.cc:237] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:00:58.969758 29507 raft_consensus.cc:2454] T 66f7d82d97b6414992f7fdcdaeb83e3d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:00:58.970192 2044 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:58.970422 2044 ts_tablet_manager.cc:1367] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.257s user 0.001s sys 0.009s
I20240501 14:00:58.971196 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:00:58.971750 2044 raft_consensus.cc:348] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.971978 2044 raft_consensus.cc:374] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.972049 2044 raft_consensus.cc:729] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.972297 2044 consensus_queue.cc:260] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:58.972898 2044 ts_tablet_manager.cc:1398] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:58.973372 2042 ts_tablet_manager.cc:1398] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240501 14:00:58.979220 29097 catalog_manager.cc:5484] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:00:58.991093 2044 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:58.992903 2044 tablet_bootstrap.cc:654] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.022248 1527 raft_consensus.cc:482] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.022495 1527 raft_consensus.cc:504] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.028515 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db862a636e5a427085a2fa31f6fe8050" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:00:59.028774 31965 raft_consensus.cc:2454] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:59.030602 29312 leader_election.cc:302] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:59.030825 2044 tablet_bootstrap.cc:492] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:59.031011 2044 ts_tablet_manager.cc:1367] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.016s
I20240501 14:00:59.031227 1968 raft_consensus.cc:2781] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.031322 1968 raft_consensus.cc:482] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.031442 1968 raft_consensus.cc:3037] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.031939 2044 raft_consensus.cc:348] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.032142 2044 raft_consensus.cc:374] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.032230 2044 raft_consensus.cc:729] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.032440 2044 consensus_queue.cc:260] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.033041 2044 ts_tablet_manager.cc:1398] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:59.034498 1968 raft_consensus.cc:504] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.036991 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db862a636e5a427085a2fa31f6fe8050" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:00:59.037204 31964 raft_consensus.cc:3037] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.042356 31964 raft_consensus.cc:2454] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:59.044726 29312 leader_election.cc:302] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:00:59.058588 1527 leader_election.cc:288] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:59.058871 1527 raft_consensus.cc:2781] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.059010 1527 raft_consensus.cc:686] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:00:59.059394 1527 consensus_queue.cc:237] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.060732 32225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db862a636e5a427085a2fa31f6fe8050" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:59.061024 32225 raft_consensus.cc:2454] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:00:59.064201 1968 leader_election.cc:288] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:59.064601 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db862a636e5a427085a2fa31f6fe8050" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:00:59.064801 32228 raft_consensus.cc:3037] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.072201 32228 raft_consensus.cc:2454] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:00:59.083716 2021 raft_consensus.cc:482] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.084039 2021 raft_consensus.cc:504] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.084707 2021 leader_election.cc:288] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:59.085340 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c943f9962fd44e0a3805a9d8e53c0a0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:00:59.085579 32360 raft_consensus.cc:2454] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:59.086356 32031 leader_election.cc:302] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:59.087335 2021 raft_consensus.cc:2781] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.087471 2021 raft_consensus.cc:482] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.087739 2021 raft_consensus.cc:3037] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.090504 2021 raft_consensus.cc:504] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.092481 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c943f9962fd44e0a3805a9d8e53c0a0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:00:59.092746 32360 raft_consensus.cc:3037] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.086001 32225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c943f9962fd44e0a3805a9d8e53c0a0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:00:59.093883 32225 raft_consensus.cc:2454] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:00:59.095417 2021 leader_election.cc:288] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:00:59.096021 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c943f9962fd44e0a3805a9d8e53c0a0" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
W20240501 14:00:59.096159 29312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 51476.5us
I20240501 14:00:59.096545 32360 raft_consensus.cc:2454] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:59.097231 32031 leader_election.cc:302] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:00:59.097501 2021 raft_consensus.cc:2781] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.097558 32228 raft_consensus.cc:3037] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.097652 2021 raft_consensus.cc:686] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:00:59.097932 2021 consensus_queue.cc:237] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.100296 32228 raft_consensus.cc:2454] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:00:59.102559 29096 catalog_manager.cc:5484] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.107357 29097 catalog_manager.cc:5484] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.153728 29636 consensus_queue.cc:237] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.155825 29377 raft_consensus.cc:1261] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:59.155822 32228 raft_consensus.cc:1261] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:59.156448 1937 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:59.157001 1548 consensus_queue.cc:1035] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:59.159849 1937 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.163921 29377 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.169729 29097 catalog_manager.cc:5484] T a82cbc0696fe4a8a815eaf0538b5aad8 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.175091 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet a82cbc0696fe4a8a815eaf0538b5aad8 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:00:59.176928 2106 tablet_replica.cc:330] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:00:59.177173 2106 raft_consensus.cc:2229] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.177690 2106 raft_consensus.cc:2258] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.178613 2106 ts_tablet_manager.cc:1875] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:59.181491 32228 raft_consensus.cc:2932] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:00:59.190824 2106 ts_tablet_manager.cc:1888] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:00:59.191036 2106 log.cc:1196] T a82cbc0696fe4a8a815eaf0538b5aad8 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/a82cbc0696fe4a8a815eaf0538b5aad8
I20240501 14:00:59.192308 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet a82cbc0696fe4a8a815eaf0538b5aad8 (table rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]) successfully deleted
I20240501 14:00:59.240185 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:59.247954 32228 raft_consensus.cc:1261] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:59.248586 1823 consensus_queue.cc:1035] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.249169 32360 raft_consensus.cc:1261] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:59.249745 1823 consensus_queue.cc:1035] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.266162 29096 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]). Expected version 1 got 0
I20240501 14:00:59.276367 2125 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.277050 29096 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]). Expected version 1 got 0
I20240501 14:00:59.277585 2125 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:59.295307 2021 consensus_queue.cc:1035] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.298609 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8] alter complete (version 1)
I20240501 14:00:59.300158 29097 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]). Expected version 1 got 0
I20240501 14:00:59.304396 1823 consensus_queue.cc:1035] T 054b34b2b424475ca4310b6dd9d97e8e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.295944 2126 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.329002 2126 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:59.330283 2128 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.330821 2128 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:59.335613 29097 catalog_manager.cc:5576] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615) does not have the latest schema for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8]). Expected version 1 got 0
I20240501 14:00:59.412568 1660 consensus_queue.cc:1035] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.422863 1660 consensus_queue.cc:1035] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.425597 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:59.436671 2125 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.437160 2125 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:00:59.438009 2128 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.438438 2128 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: MemRowSet was empty: no flush needed.
I20240501 14:00:59.438840 2126 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.439260 2126 tablet.cc:1620] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:00:59.449863 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f6331599693346aa9e1eaaff39bacbf8] alter complete (version 2)
I20240501 14:00:59.551877 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:59.566943 2125 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:59.570709 2126 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 14:00:59.571259 32162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 72204us
I20240501 14:00:59.571993 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:59.572355 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:00:59.575353 1527 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.585034 29102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=f6331599693346aa9e1eaaff39bacbf8] alter complete (version 3)
I20240501 14:00:59.597523 2128 tablet.cc:1722] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:59.618912 32077 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:00:59.624792 1527 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.640596 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:59.648617 32342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:59.660815 31944 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:59.663985 29356 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:59.685614 32206 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:59.687211 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:59.692001 29102 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:59.692209 29102 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:59.696280 29102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 8c943f9962fd44e0a3805a9d8e53c0a0
I20240501 14:00:59.697467 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:00:59.698208 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:41308
I20240501 14:00:59.717613 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 9c859b6e5b674245965fac9c113664b9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.717665 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 9c859b6e5b674245965fac9c113664b9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.718103 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c859b6e5b674245965fac9c113664b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.718125 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c859b6e5b674245965fac9c113664b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.722204 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 9c859b6e5b674245965fac9c113664b9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.722673 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c859b6e5b674245965fac9c113664b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.723731 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 0132590f8ed5423f983c53d9c19f95fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.724135 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0132590f8ed5423f983c53d9c19f95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.724285 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 0132590f8ed5423f983c53d9c19f95fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.724741 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0132590f8ed5423f983c53d9c19f95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.727857 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 6b83cf8da83047f8aa304e04d58771a8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.728314 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b83cf8da83047f8aa304e04d58771a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.733626 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 6b83cf8da83047f8aa304e04d58771a8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.734076 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b83cf8da83047f8aa304e04d58771a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.739565 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c943f9962fd44e0a3805a9d8e53c0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:00:59.746423 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 6b83cf8da83047f8aa304e04d58771a8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.746978 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b83cf8da83047f8aa304e04d58771a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.755489 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 0132590f8ed5423f983c53d9c19f95fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.755925 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0132590f8ed5423f983c53d9c19f95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.764379 2197 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:00:59.771265 2197 tablet_bootstrap.cc:654] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.800949 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:59.820171 32074 tablet_service.cc:1467] Processing CreateTablet for tablet ed6269e2b984483b9507ba6e45de69f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]), partition=
I20240501 14:00:59.820688 32074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6269e2b984483b9507ba6e45de69f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.822736 2189 tablet_replica.cc:330] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:00:59.822981 2189 raft_consensus.cc:2229] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:59.823338 2189 raft_consensus.cc:2258] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.824090 2190 tablet_replica.cc:330] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:00:59.824299 2190 raft_consensus.cc:2229] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.824525 2190 raft_consensus.cc:2258] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.825385 2193 tablet_replica.cc:330] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:00:59.825685 2193 raft_consensus.cc:2229] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.826006 2193 raft_consensus.cc:2258] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.826071 31943 tablet_service.cc:1467] Processing CreateTablet for tablet ed6269e2b984483b9507ba6e45de69f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]), partition=
I20240501 14:00:59.826445 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6269e2b984483b9507ba6e45de69f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.826833 2195 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:59.828629 2195 tablet_bootstrap.cc:654] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.830530 2192 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:59.832388 2192 tablet_bootstrap.cc:654] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.833389 2194 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:00:59.834898 2194 tablet_bootstrap.cc:654] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.838829 29616 tablet_service.cc:1467] Processing CreateTablet for tablet ed6269e2b984483b9507ba6e45de69f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]), partition=
I20240501 14:00:59.839267 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6269e2b984483b9507ba6e45de69f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.841737 2193 ts_tablet_manager.cc:1875] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.845515 2189 ts_tablet_manager.cc:1875] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.852520 2189 ts_tablet_manager.cc:1888] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.861033 2198 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:59.863806 2190 ts_tablet_manager.cc:1875] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.866564 2196 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:59.868489 2196 tablet_bootstrap.cc:654] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.871575 2190 ts_tablet_manager.cc:1888] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.878908 2198 tablet_bootstrap.cc:654] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.881985 2195 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:59.893072 2195 ts_tablet_manager.cc:1367] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.002s
I20240501 14:00:59.894194 2195 raft_consensus.cc:348] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.894667 2195 raft_consensus.cc:374] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.894868 2195 raft_consensus.cc:729] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.895169 2195 consensus_queue.cc:260] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.895855 2195 ts_tablet_manager.cc:1398] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:59.903455 2194 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:00:59.903738 2194 ts_tablet_manager.cc:1367] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20240501 14:00:59.905117 2194 raft_consensus.cc:348] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.905483 2194 raft_consensus.cc:374] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.905615 2194 raft_consensus.cc:729] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.905994 2194 consensus_queue.cc:260] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.912997 2197 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:00:59.913214 2194 ts_tablet_manager.cc:1398] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:59.913257 2197 ts_tablet_manager.cc:1367] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.149s user 0.030s sys 0.057s
I20240501 14:00:59.914175 2197 raft_consensus.cc:348] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.914482 2197 raft_consensus.cc:374] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.914613 2197 raft_consensus.cc:729] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.914913 2197 consensus_queue.cc:260] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.916620 2197 ts_tablet_manager.cc:1398] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:59.918560 2189 log.cc:1196] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/8c943f9962fd44e0a3805a9d8e53c0a0
I20240501 14:00:59.887033 2191 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:59.919260 2189 ts_tablet_manager.cc:1909] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:00:59.920579 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table_new_name [id=f6331599693346aa9e1eaaff39bacbf8]) successfully deleted
I20240501 14:00:59.920713 2195 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:00:59.920756 2191 tablet_bootstrap.cc:654] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.922739 2195 tablet_bootstrap.cc:654] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.923013 2176 raft_consensus.cc:482] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.923246 2191 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:59.923161 2176 raft_consensus.cc:504] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.923391 2191 ts_tablet_manager.cc:1367] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.004s
I20240501 14:00:59.924082 2176 leader_election.cc:288] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:59.924307 2191 raft_consensus.cc:348] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.922318 2196 tablet_bootstrap.cc:492] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:00:59.924729 2196 ts_tablet_manager.cc:1367] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.000s
I20240501 14:00:59.924844 2191 raft_consensus.cc:374] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.925012 2191 raft_consensus.cc:729] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.925251 2191 consensus_queue.cc:260] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.924894 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b83cf8da83047f8aa304e04d58771a8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:00:59.926501 29245 raft_consensus.cc:2454] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:00:59.925184 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b83cf8da83047f8aa304e04d58771a8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:00:59.927887 2196 raft_consensus.cc:348] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.928086 2196 raft_consensus.cc:374] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.928490 2191 ts_tablet_manager.cc:1398] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240501 14:00:59.929162 32161 leader_election.cc:341] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:59.929402 32161 leader_election.cc:302] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:00:59.929692 2196 raft_consensus.cc:729] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.929889 2191 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:00:59.929874 2176 raft_consensus.cc:2781] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.929992 2176 raft_consensus.cc:482] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.930084 2176 raft_consensus.cc:3037] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.931957 2191 tablet_bootstrap.cc:654] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.932097 2196 consensus_queue.cc:260] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.916196 2193 ts_tablet_manager.cc:1888] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.933151 2193 log.cc:1196] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/8c943f9962fd44e0a3805a9d8e53c0a0
I20240501 14:00:59.934008 2193 ts_tablet_manager.cc:1909] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:00:59.935124 2190 log.cc:1196] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/8c943f9962fd44e0a3805a9d8e53c0a0
I20240501 14:00:59.935673 2190 ts_tablet_manager.cc:1909] T 8c943f9962fd44e0a3805a9d8e53c0a0 P 6edd6a3430d44c38a19c2804e555059f: Deleting consensus metadata
I20240501 14:00:59.932657 2176 raft_consensus.cc:504] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.937233 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table_new_name [id=f6331599693346aa9e1eaaff39bacbf8]) successfully deleted
I20240501 14:00:59.915831 2192 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:59.937559 2192 ts_tablet_manager.cc:1367] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.003s
I20240501 14:00:59.938019 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b83cf8da83047f8aa304e04d58771a8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:00:59.938239 29245 raft_consensus.cc:3037] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.938447 2192 raft_consensus.cc:348] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.938656 2192 raft_consensus.cc:374] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.938759 2192 raft_consensus.cc:729] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.939357 2198 tablet_bootstrap.cc:492] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:59.939635 2198 ts_tablet_manager.cc:1367] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.005s
I20240501 14:00:59.940531 2198 raft_consensus.cc:348] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.940764 2198 raft_consensus.cc:374] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.940863 2198 raft_consensus.cc:729] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.941140 29245 raft_consensus.cc:2454] T 6b83cf8da83047f8aa304e04d58771a8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:00:59.941143 2198 consensus_queue.cc:260] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.940594 2192 consensus_queue.cc:260] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:00:59.943894 2198 ts_tablet_manager.cc:1398] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:59.943976 2192 ts_tablet_manager.cc:1398] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:59.944061 2196 ts_tablet_manager.cc:1398] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:00:59.944290 2192 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:00:59.945247 2196 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:00:59.945789 32161 leader_election.cc:302] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:00:59.946135 2183 raft_consensus.cc:2781] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.946275 2183 raft_consensus.cc:686] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:00:59.946635 2183 consensus_queue.cc:237] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:00:59.948076 2176 leader_election.cc:288] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:00:59.952781 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b83cf8da83047f8aa304e04d58771a8" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:00:59.953171 29636 raft_consensus.cc:3037] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.953749 2192 tablet_bootstrap.cc:654] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.954095 2198 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:00:59.956956 2196 tablet_bootstrap.cc:654] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.959355 2198 tablet_bootstrap.cc:654] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.967303 2195 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:00:59.967608 2195 ts_tablet_manager.cc:1367] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20240501 14:00:59.969099 2195 raft_consensus.cc:348] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.969558 2195 raft_consensus.cc:374] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.969699 2195 raft_consensus.cc:729] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.970050 2195 consensus_queue.cc:260] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.970948 2195 ts_tablet_manager.cc:1398] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:59.978663 2192 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:00:59.978971 2192 ts_tablet_manager.cc:1367] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20240501 14:00:59.981129 2192 raft_consensus.cc:348] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.981439 2192 raft_consensus.cc:374] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.981645 2192 raft_consensus.cc:729] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.981912 2192 consensus_queue.cc:260] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:00:59.982547 2192 ts_tablet_manager.cc:1398] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:59.993453 2191 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:00:59.993921 2198 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:00:59.993706 29102 catalog_manager.cc:5484] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:00:59.994254 2198 ts_tablet_manager.cc:1367] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.007s
I20240501 14:00:59.994468 2191 ts_tablet_manager.cc:1367] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.004s
I20240501 14:01:00.000613 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 8c943f9962fd44e0a3805a9d8e53c0a0 (table rebalancer_dynamic_table_new_name [id=f6331599693346aa9e1eaaff39bacbf8]) successfully deleted
I20240501 14:01:00.001417 2198 raft_consensus.cc:348] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.001708 2191 raft_consensus.cc:348] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.002259 2198 raft_consensus.cc:374] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.002415 2191 raft_consensus.cc:374] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.003324 2198 raft_consensus.cc:729] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.003554 2191 raft_consensus.cc:729] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.003930 2191 consensus_queue.cc:260] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.004410 2198 consensus_queue.cc:260] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.011175 2198 ts_tablet_manager.cc:1398] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240501 14:01:00.011675 2191 ts_tablet_manager.cc:1398] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:01:00.012758 2196 tablet_bootstrap.cc:492] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:00.012898 2196 ts_tablet_manager.cc:1367] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.008s
I20240501 14:01:00.013653 2196 raft_consensus.cc:348] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.013875 2196 raft_consensus.cc:374] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.013919 2196 raft_consensus.cc:729] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.014139 2196 consensus_queue.cc:260] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.015086 2196 ts_tablet_manager.cc:1398] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:00.028352 29636 raft_consensus.cc:2454] T 6b83cf8da83047f8aa304e04d58771a8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:00.049914 2021 raft_consensus.cc:482] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.050143 2021 raft_consensus.cc:504] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.050959 2021 leader_election.cc:288] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:00.052754 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6269e2b984483b9507ba6e45de69f8" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:00.053054 31965 raft_consensus.cc:2454] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:00.053501 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6269e2b984483b9507ba6e45de69f8" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:00.053736 29636 raft_consensus.cc:2454] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:00.053740 32032 leader_election.cc:302] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:00.054597 2021 raft_consensus.cc:2781] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.054690 2021 raft_consensus.cc:482] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.054785 2021 raft_consensus.cc:3037] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.060438 2021 raft_consensus.cc:504] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.061761 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6269e2b984483b9507ba6e45de69f8" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:00.061956 29636 raft_consensus.cc:3037] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.064659 29636 raft_consensus.cc:2454] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:00.066661 32030 leader_election.cc:302] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:00.066967 1823 raft_consensus.cc:2781] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.069231 1823 raft_consensus.cc:686] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:00.069518 1823 consensus_queue.cc:237] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.072614 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6269e2b984483b9507ba6e45de69f8" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:00.072856 31965 raft_consensus.cc:3037] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.074074 2021 leader_election.cc:288] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:00.076471 31965 raft_consensus.cc:2454] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:00.096529 29102 catalog_manager.cc:5484] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:00.106784 1823 raft_consensus.cc:482] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.106976 1823 raft_consensus.cc:504] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:00.107846 1823 leader_election.cc:288] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:00.108691 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0132590f8ed5423f983c53d9c19f95fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:00.108777 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0132590f8ed5423f983c53d9c19f95fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:00.108954 32362 raft_consensus.cc:2454] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:00.109066 32228 raft_consensus.cc:2454] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:00.109730 32031 leader_election.cc:302] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:00.110771 1823 raft_consensus.cc:2781] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.110922 1823 raft_consensus.cc:482] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.111032 1823 raft_consensus.cc:3037] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.114974 1823 raft_consensus.cc:504] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:00.116601 1823 leader_election.cc:288] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:00.118255 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0132590f8ed5423f983c53d9c19f95fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:00.118495 32362 raft_consensus.cc:3037] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.124239 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0132590f8ed5423f983c53d9c19f95fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:00.124564 32228 raft_consensus.cc:3037] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.125734 32362 raft_consensus.cc:2454] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:00.127506 32228 raft_consensus.cc:2454] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:00.128944 32031 leader_election.cc:302] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:00.129508 1823 raft_consensus.cc:2781] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.129637 1823 raft_consensus.cc:686] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:00.129915 1823 consensus_queue.cc:237] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:00.138760 29102 catalog_manager.cc:5484] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:00.163440 1944 raft_consensus.cc:482] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.164044 1944 raft_consensus.cc:504] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:00.164850 1944 leader_election.cc:288] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:00.165572 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c859b6e5b674245965fac9c113664b9" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:00.165722 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c859b6e5b674245965fac9c113664b9" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:00.165794 31964 raft_consensus.cc:2454] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:00.165951 29377 raft_consensus.cc:2454] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:00.166163 32296 leader_election.cc:302] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:00.166424 1944 raft_consensus.cc:2781] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.166524 1944 raft_consensus.cc:482] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.166607 1944 raft_consensus.cc:3037] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.169225 1944 raft_consensus.cc:504] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:00.170357 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c859b6e5b674245965fac9c113664b9" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:00.170457 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c859b6e5b674245965fac9c113664b9" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:00.170543 29377 raft_consensus.cc:3037] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.170663 31964 raft_consensus.cc:3037] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.172538 31964 raft_consensus.cc:2454] T 9c859b6e5b674245965fac9c113664b9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:00.172902 1944 leader_election.cc:288] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:00.173002 32296 leader_election.cc:302] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:00.173252 1944 raft_consensus.cc:2781] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.173276 29377 raft_consensus.cc:2454] T 9c859b6e5b674245965fac9c113664b9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:00.173461 1944 raft_consensus.cc:686] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:00.173734 1944 consensus_queue.cc:237] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:00.178007 29099 catalog_manager.cc:5484] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:00.183866 29635 consensus_queue.cc:237] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:00.185807 29507 raft_consensus.cc:1261] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:00.185813 32096 raft_consensus.cc:1261] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:00.186286 2219 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:00.186547 1937 consensus_queue.cc:1035] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:00.189175 1548 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:00.189954 29505 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:00.190249 32096 raft_consensus.cc:2932] T e513634d27c5467fb49c9717c43904cd P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:00.194866 29099 catalog_manager.cc:5484] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:00.200289 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet e513634d27c5467fb49c9717c43904cd with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51134
I20240501 14:01:00.201035 2240 tablet_replica.cc:330] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:01:00.201467 2240 raft_consensus.cc:2229] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:00.201972 2240 raft_consensus.cc:2258] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.203565 2240 ts_tablet_manager.cc:1875] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:00.209136 2240 ts_tablet_manager.cc:1888] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:00.209350 2240 log.cc:1196] T e513634d27c5467fb49c9717c43904cd P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/e513634d27c5467fb49c9717c43904cd
I20240501 14:01:00.210520 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet e513634d27c5467fb49c9717c43904cd (table rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]) successfully deleted
I20240501 14:01:00.252090 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:00.258384 31964 raft_consensus.cc:1261] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:00.258702 29635 raft_consensus.cc:1261] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:00.259320 2021 consensus_queue.cc:1035] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.259662 1823 consensus_queue.cc:1035] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.263412 2256 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.264001 2256 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:00.265631 29099 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]). Expected version 1 got 0
I20240501 14:01:00.272166 2257 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.274233 2257 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:00.276201 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571] alter complete (version 1)
I20240501 14:01:00.282027 29099 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]). Expected version 1 got 0
I20240501 14:01:00.291834 29101 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]). Expected version 1 got 0
I20240501 14:01:00.296455 29101 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571]). Expected version 1 got 0
I20240501 14:01:00.294395 2268 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.298285 2268 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:00.381855 29101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:00.390511 2256 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.391089 2256 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:00.393694 2268 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.394220 2268 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:00.407734 2257 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.408237 2257 tablet.cc:1620] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:00.408864 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=ee582aed3d334599b5c199d4d3503571] alter complete (version 2)
I20240501 14:01:00.438661 2176 consensus_queue.cc:1035] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.464915 32076 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:01:00.486450 2183 consensus_queue.cc:1035] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.487105 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:00.514813 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:00.527390 2256 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.530063 2268 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.534528 2257 tablet.cc:1722] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.538868 1823 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.560271 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=ee582aed3d334599b5c199d4d3503571] alter complete (version 3)
I20240501 14:01:00.564988 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:00.576539 2021 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.590770 32208 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:00.616374 29356 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:00.660677 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:00.664288 31944 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:00.664666 29098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:00.664956 29098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:00.668861 29098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet ed6269e2b984483b9507ba6e45de69f8
I20240501 14:01:00.670053 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet ed6269e2b984483b9507ba6e45de69f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:00.672724 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet ed6269e2b984483b9507ba6e45de69f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:00.673928 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet ed6269e2b984483b9507ba6e45de69f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:00.697741 2317 tablet_replica.cc:330] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:00.698895 2318 tablet_replica.cc:330] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:00.699819 2317 raft_consensus.cc:2229] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:00.700884 2317 raft_consensus.cc:2258] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.702575 2318 raft_consensus.cc:2229] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:00.702803 2318 raft_consensus.cc:2258] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.703948 2317 ts_tablet_manager.cc:1875] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.708343 2318 ts_tablet_manager.cc:1875] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.710800 2316 tablet_replica.cc:330] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:00.721200 2316 raft_consensus.cc:2229] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:00.721563 2316 raft_consensus.cc:2258] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.722697 2316 ts_tablet_manager.cc:1875] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.724346 2318 ts_tablet_manager.cc:1888] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.724526 2318 log.cc:1196] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/ed6269e2b984483b9507ba6e45de69f8
I20240501 14:01:00.725051 2318 ts_tablet_manager.cc:1909] T ed6269e2b984483b9507ba6e45de69f8 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:00.726218 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table_new_name [id=ee582aed3d334599b5c199d4d3503571]) successfully deleted
I20240501 14:01:00.728899 2317 ts_tablet_manager.cc:1888] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.729995 2317 log.cc:1196] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/ed6269e2b984483b9507ba6e45de69f8
I20240501 14:01:00.730475 2317 ts_tablet_manager.cc:1909] T ed6269e2b984483b9507ba6e45de69f8 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:00.731985 1944 consensus_queue.cc:1035] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.733772 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table_new_name [id=ee582aed3d334599b5c199d4d3503571]) successfully deleted
I20240501 14:01:00.736588 2316 ts_tablet_manager.cc:1888] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.738565 32342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:00.740115 2316 log.cc:1196] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed6269e2b984483b9507ba6e45de69f8
I20240501 14:01:00.740509 2316 ts_tablet_manager.cc:1909] T ed6269e2b984483b9507ba6e45de69f8 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:00.745146 1944 consensus_queue.cc:1035] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.747610 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet ed6269e2b984483b9507ba6e45de69f8 (table rebalancer_dynamic_table_new_name [id=ee582aed3d334599b5c199d4d3503571]) successfully deleted
I20240501 14:01:00.770954 29099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:00.774019 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:00.785598 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 288ef316c7df47faada178a425f0ff01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]), partition=
I20240501 14:01:00.785740 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 288ef316c7df47faada178a425f0ff01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]), partition=
I20240501 14:01:00.786041 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288ef316c7df47faada178a425f0ff01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.786134 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288ef316c7df47faada178a425f0ff01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.792007 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 288ef316c7df47faada178a425f0ff01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]), partition=
I20240501 14:01:00.792402 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288ef316c7df47faada178a425f0ff01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.792316 32342 tablet_service.cc:1467] Processing CreateTablet for tablet eb982ed7bcf74eb981c5199d87f9b4c4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.792759 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb982ed7bcf74eb981c5199d87f9b4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.793161 31944 tablet_service.cc:1467] Processing CreateTablet for tablet eb982ed7bcf74eb981c5199d87f9b4c4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.793653 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb982ed7bcf74eb981c5199d87f9b4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.795747 31943 tablet_service.cc:1467] Processing CreateTablet for tablet ba3bd81448244b5bbdcd9eb8d23aee03 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.796190 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3bd81448244b5bbdcd9eb8d23aee03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.796762 32076 tablet_service.cc:1467] Processing CreateTablet for tablet eb982ed7bcf74eb981c5199d87f9b4c4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.797220 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb982ed7bcf74eb981c5199d87f9b4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.801684 32206 tablet_service.cc:1467] Processing CreateTablet for tablet ba3bd81448244b5bbdcd9eb8d23aee03 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.801784 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 815ce94772f2454ea755d33dd04a87d2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.801944 32074 tablet_service.cc:1467] Processing CreateTablet for tablet ba3bd81448244b5bbdcd9eb8d23aee03 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.802210 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3bd81448244b5bbdcd9eb8d23aee03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.802260 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815ce94772f2454ea755d33dd04a87d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.803181 32074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3bd81448244b5bbdcd9eb8d23aee03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.805433 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 815ce94772f2454ea755d33dd04a87d2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.805819 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815ce94772f2454ea755d33dd04a87d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.814489 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 815ce94772f2454ea755d33dd04a87d2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.815088 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815ce94772f2454ea755d33dd04a87d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.847738 2329 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:00.849619 2329 tablet_bootstrap.cc:654] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.850189 2328 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:00.852048 2328 tablet_bootstrap.cc:654] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.853799 2330 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:00.854586 2329 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:00.855015 2329 ts_tablet_manager.cc:1367] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:01:00.855410 2330 tablet_bootstrap.cc:654] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.855715 2329 raft_consensus.cc:348] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.855892 2329 raft_consensus.cc:374] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.856002 2329 raft_consensus.cc:729] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.856235 2329 consensus_queue.cc:260] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.856921 2329 ts_tablet_manager.cc:1398] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:00.858297 2330 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:00.858443 2330 ts_tablet_manager.cc:1367] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:00.859349 2327 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:00.860195 2328 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:00.860427 2328 ts_tablet_manager.cc:1367] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:01:00.863369 2328 raft_consensus.cc:348] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.863751 2328 raft_consensus.cc:374] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.864017 2328 raft_consensus.cc:729] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.864336 2328 consensus_queue.cc:260] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.865057 2328 ts_tablet_manager.cc:1398] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:00.867908 2328 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:00.869418 2328 tablet_bootstrap.cc:654] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.872366 2326 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:00.874653 2326 tablet_bootstrap.cc:654] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.880096 2328 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:00.880275 2328 ts_tablet_manager.cc:1367] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:01:00.881471 2325 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:00.882467 2328 raft_consensus.cc:348] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.882737 2328 raft_consensus.cc:374] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.882817 2328 raft_consensus.cc:729] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.883075 2328 consensus_queue.cc:260] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.883710 2328 ts_tablet_manager.cc:1398] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:00.883718 2325 tablet_bootstrap.cc:654] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.884183 2328 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:00.885406 2330 raft_consensus.cc:348] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.885704 2330 raft_consensus.cc:374] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.885841 2330 raft_consensus.cc:729] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.886109 2330 consensus_queue.cc:260] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.888017 2328 tablet_bootstrap.cc:654] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.888391 2330 ts_tablet_manager.cc:1398] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.888814 2330 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:00.890537 2330 tablet_bootstrap.cc:654] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.891582 2298 raft_consensus.cc:482] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.891789 2298 raft_consensus.cc:504] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.892782 2298 leader_election.cc:288] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:00.893747 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:01:00.894351 32162 leader_election.cc:341] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:00.897013 2327 tablet_bootstrap.cc:654] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.902365 2326 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:00.902719 2326 ts_tablet_manager.cc:1367] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 14:01:00.903435 2325 tablet_bootstrap.cc:492] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:00.903643 2325 ts_tablet_manager.cc:1367] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:01:00.904035 2326 raft_consensus.cc:348] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.904311 2326 raft_consensus.cc:374] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.904458 2326 raft_consensus.cc:729] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.904440 2325 raft_consensus.cc:348] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.904762 2326 consensus_queue.cc:260] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.905586 2325 raft_consensus.cc:374] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.905718 2325 raft_consensus.cc:729] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.906922 2326 ts_tablet_manager.cc:1398] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.907191 2326 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:00.906242 2325 consensus_queue.cc:260] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:00.914562 2326 tablet_bootstrap.cc:654] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.893447 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:01:00.915194 32163 leader_election.cc:341] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:00.915377 32163 leader_election.cc:302] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:00.915696 2298 raft_consensus.cc:2726] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:00.917583 2325 ts_tablet_manager.cc:1398] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:00.922719 2327 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:00.922986 2327 ts_tablet_manager.cc:1367] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.002s
I20240501 14:01:00.923885 2327 raft_consensus.cc:348] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.924194 2327 raft_consensus.cc:374] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.924320 2327 raft_consensus.cc:729] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.924669 2327 consensus_queue.cc:260] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.925406 2327 ts_tablet_manager.cc:1398] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:00.927274 2327 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:00.929111 2327 tablet_bootstrap.cc:654] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.938071 2330 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:00.938356 2330 ts_tablet_manager.cc:1367] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.002s
I20240501 14:01:00.938566 2328 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:00.938800 2328 ts_tablet_manager.cc:1367] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.013s
I20240501 14:01:00.939257 2330 raft_consensus.cc:348] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.939642 2328 raft_consensus.cc:348] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.940605 2330 raft_consensus.cc:374] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.940753 2330 raft_consensus.cc:729] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.940907 2328 raft_consensus.cc:374] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.941336 2328 raft_consensus.cc:729] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.941183 2330 consensus_queue.cc:260] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.941617 2328 consensus_queue.cc:260] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:00.942167 2330 ts_tablet_manager.cc:1398] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.942670 2330 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:00.946118 2330 tablet_bootstrap.cc:654] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.952471 2328 ts_tablet_manager.cc:1398] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:00.954190 2327 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:00.954435 2327 ts_tablet_manager.cc:1367] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20240501 14:01:00.955219 2327 raft_consensus.cc:348] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.955514 2327 raft_consensus.cc:374] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.955633 2327 raft_consensus.cc:729] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.955955 2327 consensus_queue.cc:260] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.959558 2327 ts_tablet_manager.cc:1398] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:01:00.960036 2326 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:00.960258 2326 ts_tablet_manager.cc:1367] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.000s
I20240501 14:01:00.961102 2326 raft_consensus.cc:348] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.961392 2326 raft_consensus.cc:374] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.961508 2326 raft_consensus.cc:729] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.961822 2326 consensus_queue.cc:260] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.965996 2326 ts_tablet_manager.cc:1398] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:00.966305 2330 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:00.966465 2330 ts_tablet_manager.cc:1367] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20240501 14:01:00.967187 2330 raft_consensus.cc:348] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.967357 2330 raft_consensus.cc:374] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.967428 2330 raft_consensus.cc:729] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.967644 2330 consensus_queue.cc:260] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:00.968204 2330 ts_tablet_manager.cc:1398] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:01.002259 1823 raft_consensus.cc:482] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.002485 1823 raft_consensus.cc:504] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.003167 1823 leader_election.cc:288] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:01.003829 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:01.004112 32227 raft_consensus.cc:2454] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:01.004087 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:01.004293 32360 raft_consensus.cc:2454] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:01.004725 32031 leader_election.cc:302] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:01.005453 1823 raft_consensus.cc:2781] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.005545 1823 raft_consensus.cc:482] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.005641 1823 raft_consensus.cc:3037] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.008585 1823 raft_consensus.cc:504] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.009651 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:01.009904 32227 raft_consensus.cc:3037] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.010125 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815ce94772f2454ea755d33dd04a87d2" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:01.010332 32360 raft_consensus.cc:3037] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.012765 32227 raft_consensus.cc:2454] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:01.013406 32031 leader_election.cc:302] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:01.013664 32360 raft_consensus.cc:2454] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:01.014189 1823 leader_election.cc:288] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:01.014400 2021 raft_consensus.cc:2781] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.014564 2021 raft_consensus.cc:686] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:01.014799 2021 consensus_queue.cc:237] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.035341 29098 catalog_manager.cc:5484] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:01.057150 2021 raft_consensus.cc:482] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.057322 2021 raft_consensus.cc:504] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.057938 2021 leader_election.cc:288] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:01.058677 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb982ed7bcf74eb981c5199d87f9b4c4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:01.058960 32360 raft_consensus.cc:2454] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:01.059227 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb982ed7bcf74eb981c5199d87f9b4c4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:01.059453 32031 leader_election.cc:302] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:01.059492 31965 raft_consensus.cc:2454] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:01.059756 2021 raft_consensus.cc:2781] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.059895 2021 raft_consensus.cc:482] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.059999 2021 raft_consensus.cc:3037] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.061885 2298 raft_consensus.cc:482] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.062057 2298 raft_consensus.cc:504] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:01.062700 2298 leader_election.cc:288] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:01.062769 2021 raft_consensus.cc:504] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.063460 2021 leader_election.cc:288] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:01.063975 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb982ed7bcf74eb981c5199d87f9b4c4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:01.064221 32360 raft_consensus.cc:3037] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.064595 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288ef316c7df47faada178a425f0ff01" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:01.064821 29245 raft_consensus.cc:2454] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:01.064949 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb982ed7bcf74eb981c5199d87f9b4c4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:01.065192 31965 raft_consensus.cc:3037] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.067209 32360 raft_consensus.cc:2454] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:01.067785 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288ef316c7df47faada178a425f0ff01" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:01.068038 29506 raft_consensus.cc:2454] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:01.068308 32031 leader_election.cc:302] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:01.068662 2021 raft_consensus.cc:2781] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.069239 2021 raft_consensus.cc:686] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:01.069490 32161 leader_election.cc:302] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:01.069828 2298 raft_consensus.cc:2781] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.069948 2298 raft_consensus.cc:482] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.070039 2298 raft_consensus.cc:3037] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.069944 2021 consensus_queue.cc:237] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.070951 31965 raft_consensus.cc:2454] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:01.072755 2298 raft_consensus.cc:504] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:01.073436 2298 leader_election.cc:288] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:01.074206 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288ef316c7df47faada178a425f0ff01" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:01.074435 29506 raft_consensus.cc:3037] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.074401 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288ef316c7df47faada178a425f0ff01" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:01.074667 29245 raft_consensus.cc:3037] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.077862 29245 raft_consensus.cc:2454] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:01.078326 32161 leader_election.cc:302] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:01.078684 2298 raft_consensus.cc:2781] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.078939 2298 raft_consensus.cc:686] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:01.079249 2298 consensus_queue.cc:237] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:01.080729 29506 raft_consensus.cc:2454] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:01.084379 29098 catalog_manager.cc:5484] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:01.095767 29098 catalog_manager.cc:5484] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:01.116401 2259 raft_consensus.cc:482] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.116608 2259 raft_consensus.cc:504] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.117703 2259 leader_election.cc:288] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:01.117787 29245 consensus_queue.cc:237] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:01.119100 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3bd81448244b5bbdcd9eb8d23aee03" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:01.119338 32097 raft_consensus.cc:2454] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:01.119397 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3bd81448244b5bbdcd9eb8d23aee03" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:01.119602 32227 raft_consensus.cc:2454] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:01.119809 31900 leader_election.cc:302] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:01.120091 2259 raft_consensus.cc:2781] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.120196 2259 raft_consensus.cc:482] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.120273 2259 raft_consensus.cc:3037] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.123121 2259 raft_consensus.cc:504] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.123301 29377 raft_consensus.cc:1261] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:01.124011 32097 raft_consensus.cc:1261] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:01.124123 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3bd81448244b5bbdcd9eb8d23aee03" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:01.124334 32096 raft_consensus.cc:3037] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.124342 1939 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:01.124629 2289 consensus_queue.cc:1035] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:01.125324 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3bd81448244b5bbdcd9eb8d23aee03" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:01.125535 32227 raft_consensus.cc:3037] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.127270 32096 raft_consensus.cc:2454] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:01.128350 32227 raft_consensus.cc:2454] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:01.129431 31899 leader_election.cc:302] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:01.131711 2259 leader_election.cc:288] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:01.132349 2259 raft_consensus.cc:2781] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.132508 2259 raft_consensus.cc:686] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:01.132897 2259 consensus_queue.cc:237] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.153900 1939 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:01.158775 32096 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:01.164906 29098 catalog_manager.cc:5484] T 1a412c988fd946d7b7d51606be01d1fa P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:01.174464 32340 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a412c988fd946d7b7d51606be01d1fa with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:01.181468 2374 tablet_replica.cc:330] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:01.181918 2374 raft_consensus.cc:2229] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.181828 29376 raft_consensus.cc:2932] T 1a412c988fd946d7b7d51606be01d1fa P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:01.182579 2374 raft_consensus.cc:2258] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.183264 2374 ts_tablet_manager.cc:1875] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:01.192211 2374 ts_tablet_manager.cc:1888] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:01.192518 2374 log.cc:1196] T 1a412c988fd946d7b7d51606be01d1fa P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/1a412c988fd946d7b7d51606be01d1fa
I20240501 14:01:01.193639 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 1a412c988fd946d7b7d51606be01d1fa (table rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]) successfully deleted
W20240501 14:01:01.196125 29442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 55540.1us
I20240501 14:01:01.198201 29098 catalog_manager.cc:5484] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:01.258441 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:01.265355 29506 raft_consensus.cc:1261] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.265844 29245 raft_consensus.cc:1261] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.266763 2298 consensus_queue.cc:1035] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.267306 2183 consensus_queue.cc:1035] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.286486 2402 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.287647 2402 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:01.291103 29098 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]). Expected version 1 got 0
I20240501 14:01:01.292392 29105 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]). Expected version 1 got 0
I20240501 14:01:01.294456 2403 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.295339 2403 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:01.297215 29105 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]). Expected version 1 got 0
I20240501 14:01:01.302227 2405 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.302902 29099 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503]). Expected version 1 got 0
I20240501 14:01:01.302985 2405 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:01.307585 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503] alter complete (version 1)
I20240501 14:01:01.418555 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:01.433532 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:01.439904 32076 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:01.443341 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:01.467171 2213 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.483580 2402 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.484102 2402 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:01.486305 2405 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.486871 2405 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:01.495174 2403 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.495599 29098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=1592c41d2f9e420693f057cd11103503] alter complete (version 2)
I20240501 14:01:01.495700 2403 tablet.cc:1620] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:01.503252 2213 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.507529 32340 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:01.545637 2021 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.549785 32207 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:01.557273 1823 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.603013 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:01.613224 2402 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.615272 2403 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.615880 2405 tablet.cc:1722] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.620564 2369 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.625766 29101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=1592c41d2f9e420693f057cd11103503] alter complete (version 3)
I20240501 14:01:01.631567 31943 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:01.648484 29356 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:01.651628 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:01.664453 2259 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.741504 29101 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:01.741741 29101 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:01.745087 29101 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 288ef316c7df47faada178a425f0ff01
I20240501 14:01:01.746062 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 288ef316c7df47faada178a425f0ff01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:01.746475 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 288ef316c7df47faada178a425f0ff01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:01.746721 2452 tablet_replica.cc:330] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:01.746825 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 288ef316c7df47faada178a425f0ff01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:01.746968 2452 raft_consensus.cc:2229] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:01.747339 2452 raft_consensus.cc:2258] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.747380 2454 tablet_replica.cc:330] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:01.747601 2454 raft_consensus.cc:2229] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.747848 2454 raft_consensus.cc:2258] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.748107 2452 ts_tablet_manager.cc:1875] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.757710 2452 ts_tablet_manager.cc:1888] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.757969 2452 log.cc:1196] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/288ef316c7df47faada178a425f0ff01
I20240501 14:01:01.758553 2452 ts_tablet_manager.cc:1909] T 288ef316c7df47faada178a425f0ff01 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:01.759655 2454 ts_tablet_manager.cc:1875] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.759842 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table_new_name [id=1592c41d2f9e420693f057cd11103503]) successfully deleted
I20240501 14:01:01.767154 2454 ts_tablet_manager.cc:1888] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.767441 2454 log.cc:1196] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/288ef316c7df47faada178a425f0ff01
I20240501 14:01:01.767971 2454 ts_tablet_manager.cc:1909] T 288ef316c7df47faada178a425f0ff01 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:01:01.768162 2453 tablet_replica.cc:330] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:01.768764 2453 raft_consensus.cc:2229] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.769066 2453 raft_consensus.cc:2258] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.771231 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table_new_name [id=1592c41d2f9e420693f057cd11103503]) successfully deleted
I20240501 14:01:01.772811 2453 ts_tablet_manager.cc:1875] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.778263 2453 ts_tablet_manager.cc:1888] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.778393 2453 log.cc:1196] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/288ef316c7df47faada178a425f0ff01
I20240501 14:01:01.778780 2453 ts_tablet_manager.cc:1909] T 288ef316c7df47faada178a425f0ff01 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:01:01.779798 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 288ef316c7df47faada178a425f0ff01 (table rebalancer_dynamic_table_new_name [id=1592c41d2f9e420693f057cd11103503]) successfully deleted
I20240501 14:01:01.791239 29101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:01.803602 31943 tablet_service.cc:1467] Processing CreateTablet for tablet eb3c16421adb477e821495e785849cc4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.803699 32340 tablet_service.cc:1467] Processing CreateTablet for tablet eb3c16421adb477e821495e785849cc4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.804119 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3c16421adb477e821495e785849cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.804158 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3c16421adb477e821495e785849cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.808126 32342 tablet_service.cc:1467] Processing CreateTablet for tablet fe99c587804f47c3b58ec3ddbab8a371 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.808585 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe99c587804f47c3b58ec3ddbab8a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.812548 2465 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:01.813395 29487 tablet_service.cc:1467] Processing CreateTablet for tablet e2d4f5ec80e64aa6aceb42a0748af62b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.813884 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d4f5ec80e64aa6aceb42a0748af62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.814666 2465 tablet_bootstrap.cc:654] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.819597 29356 tablet_service.cc:1467] Processing CreateTablet for tablet eb3c16421adb477e821495e785849cc4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.819630 31944 tablet_service.cc:1467] Processing CreateTablet for tablet e2d4f5ec80e64aa6aceb42a0748af62b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.820075 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3c16421adb477e821495e785849cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.820143 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d4f5ec80e64aa6aceb42a0748af62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.819628 31943 tablet_service.cc:1467] Processing CreateTablet for tablet fe99c587804f47c3b58ec3ddbab8a371 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.825337 2468 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:01.825423 29357 tablet_service.cc:1467] Processing CreateTablet for tablet fe99c587804f47c3b58ec3ddbab8a371 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.825578 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe99c587804f47c3b58ec3ddbab8a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.825819 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe99c587804f47c3b58ec3ddbab8a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.828723 32076 tablet_service.cc:1467] Processing CreateTablet for tablet e2d4f5ec80e64aa6aceb42a0748af62b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.828949 2468 tablet_bootstrap.cc:654] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.829182 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d4f5ec80e64aa6aceb42a0748af62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.849421 29098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:01.858114 2465 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:01.859009 2465 ts_tablet_manager.cc:1367] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20240501 14:01:01.861464 2465 raft_consensus.cc:348] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.861824 2465 raft_consensus.cc:374] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.861940 2465 raft_consensus.cc:729] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.863377 2465 consensus_queue.cc:260] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:01.863802 2469 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:01.864889 2471 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:01.866436 2471 tablet_bootstrap.cc:654] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.870054 2469 tablet_bootstrap.cc:654] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.871517 2465 ts_tablet_manager.cc:1398] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:01:01.873005 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]), partition=
I20240501 14:01:01.873457 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0c49089dbc40efa388b025c7eb2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.878077 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]), partition=
I20240501 14:01:01.878436 2465 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:01.878566 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0c49089dbc40efa388b025c7eb2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.880869 2465 tablet_bootstrap.cc:654] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.881961 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]), partition=
I20240501 14:01:01.882385 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0c49089dbc40efa388b025c7eb2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.886193 2470 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:01.890187 2468 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:01.890363 2468 ts_tablet_manager.cc:1367] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.004s
I20240501 14:01:01.891219 2468 raft_consensus.cc:348] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.891450 2468 raft_consensus.cc:374] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.891533 2468 raft_consensus.cc:729] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.891775 2468 consensus_queue.cc:260] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.896339 2470 tablet_bootstrap.cc:654] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.899040 2468 ts_tablet_manager.cc:1398] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:01:01.909123 2470 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:01.911356 2470 ts_tablet_manager.cc:1367] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.003s
I20240501 14:01:01.912464 2470 raft_consensus.cc:348] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.912791 2470 raft_consensus.cc:374] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.912959 2470 raft_consensus.cc:729] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.913321 2470 consensus_queue.cc:260] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.918943 2465 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:01.919214 2465 ts_tablet_manager.cc:1367] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20240501 14:01:01.920210 2465 raft_consensus.cc:348] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.920557 2470 ts_tablet_manager.cc:1398] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:01:01.920907 2470 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:01.921200 2465 raft_consensus.cc:374] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.921535 2465 raft_consensus.cc:729] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.921694 2482 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:01.924377 2470 tablet_bootstrap.cc:654] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.924700 2465 consensus_queue.cc:260] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.927978 2482 tablet_bootstrap.cc:654] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.931133 2465 ts_tablet_manager.cc:1398] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:01:01.936370 2471 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:01.936592 2471 ts_tablet_manager.cc:1367] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.072s user 0.014s sys 0.009s
I20240501 14:01:01.936784 2469 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:01.937496 2471 raft_consensus.cc:348] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.937907 2469 ts_tablet_manager.cc:1367] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.016s
I20240501 14:01:01.938712 2469 raft_consensus.cc:348] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.939006 2471 raft_consensus.cc:374] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.939121 2471 raft_consensus.cc:729] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.939393 2471 consensus_queue.cc:260] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.939692 2469 raft_consensus.cc:374] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.939842 2469 raft_consensus.cc:729] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.940114 2469 consensus_queue.cc:260] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.940830 2471 ts_tablet_manager.cc:1398] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:01:01.941399 2469 ts_tablet_manager.cc:1398] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:01.941797 2471 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:01.943063 2469 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:01.947108 2482 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:01.947345 2482 ts_tablet_manager.cc:1367] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240501 14:01:01.948652 2482 raft_consensus.cc:348] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:01.949024 2482 raft_consensus.cc:374] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.949185 2482 raft_consensus.cc:729] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.949477 2465 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:01.949666 2482 consensus_queue.cc:260] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:01.951665 2482 ts_tablet_manager.cc:1398] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:01.952476 2471 tablet_bootstrap.cc:654] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.956141 29245 consensus_queue.cc:237] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.958266 2469 tablet_bootstrap.cc:654] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.961910 2465 tablet_bootstrap.cc:654] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.970687 31965 raft_consensus.cc:1261] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:01.970850 2465 tablet_bootstrap.cc:492] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:01.971002 2465 ts_tablet_manager.cc:1367] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20240501 14:01:01.971887 2465 raft_consensus.cc:348] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.972218 2465 raft_consensus.cc:374] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.972373 2465 raft_consensus.cc:729] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.972718 2465 consensus_queue.cc:260] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.973140 2289 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:01.975689 29507 raft_consensus.cc:1261] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:01.976486 2289 consensus_queue.cc:1035] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:01.977543 2465 ts_tablet_manager.cc:1398] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:01.979043 2450 raft_consensus.cc:482] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.979197 2450 raft_consensus.cc:504] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.980165 2450 leader_election.cc:288] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:01.980461 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d4f5ec80e64aa6aceb42a0748af62b" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:01.980708 29506 raft_consensus.cc:2454] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:01.981179 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d4f5ec80e64aa6aceb42a0748af62b" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:01.981421 32096 raft_consensus.cc:2454] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:01.981629 2339 raft_consensus.cc:482] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.981791 2339 raft_consensus.cc:504] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.981959 31900 leader_election.cc:302] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:01.982635 2339 leader_election.cc:288] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:01.984643 2450 raft_consensus.cc:2781] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.984752 2450 raft_consensus.cc:482] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.984851 2450 raft_consensus.cc:3037] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.985203 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe99c587804f47c3b58ec3ddbab8a371" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:01.985414 31964 raft_consensus.cc:2454] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:01.986137 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe99c587804f47c3b58ec3ddbab8a371" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:01.986353 32360 raft_consensus.cc:2454] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:01.986901 29312 leader_election.cc:302] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:01.987208 2340 raft_consensus.cc:2781] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.987272 2340 raft_consensus.cc:482] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.987323 2340 raft_consensus.cc:3037] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.987895 2450 raft_consensus.cc:504] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.989356 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d4f5ec80e64aa6aceb42a0748af62b" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:01.989607 32096 raft_consensus.cc:3037] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.990646 2340 raft_consensus.cc:504] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.992216 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d4f5ec80e64aa6aceb42a0748af62b" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:01.992460 29505 raft_consensus.cc:3037] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.992470 32096 raft_consensus.cc:2454] T e2d4f5ec80e64aa6aceb42a0748af62b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:01.992869 31900 leader_election.cc:302] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:01.993207 2259 raft_consensus.cc:2781] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.993340 2259 raft_consensus.cc:686] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:01.993655 2259 consensus_queue.cc:237] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:01.998122 2450 leader_election.cc:288] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:01.998569 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe99c587804f47c3b58ec3ddbab8a371" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:01.998843 32360 raft_consensus.cc:3037] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.008246 1939 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.010542 2465 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:02.014439 31965 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.021090 32360 raft_consensus.cc:2454] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:02.021857 29311 leader_election.cc:302] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:02.022302 2339 raft_consensus.cc:2781] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.023511 2340 leader_election.cc:288] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:02.026625 29507 raft_consensus.cc:2932] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.030597 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe99c587804f47c3b58ec3ddbab8a371" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:02.030879 31964 raft_consensus.cc:3037] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.041182 31964 raft_consensus.cc:2454] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:02.064254 29098 catalog_manager.cc:5484] T 8b925d954c9143279ba69ae5c90d3266 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.068905 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b925d954c9143279ba69ae5c90d3266 with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:02.085420 2500 tablet_replica.cc:330] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:02.085876 2500 raft_consensus.cc:2229] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.086360 2500 raft_consensus.cc:2258] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.087469 2500 ts_tablet_manager.cc:1875] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:02.099695 2450 raft_consensus.cc:482] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.099918 2450 raft_consensus.cc:504] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.101121 2450 leader_election.cc:288] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:02.101738 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
W20240501 14:01:02.102420 31900 leader_election.cc:341] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.106302 29098 catalog_manager.cc:5484] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.099751 2465 tablet_bootstrap.cc:654] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.115546 2339 raft_consensus.cc:686] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:01:02.115871 2339 consensus_queue.cc:237] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:02.116320 2500 ts_tablet_manager.cc:1888] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
W20240501 14:01:02.117658 32296 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 87813.4us
W20240501 14:01:02.117977 29311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 66183.1us
I20240501 14:01:02.120054 2500 log.cc:1196] T 8b925d954c9143279ba69ae5c90d3266 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b925d954c9143279ba69ae5c90d3266
I20240501 14:01:02.121578 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 8b925d954c9143279ba69ae5c90d3266 (table rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]) successfully deleted
I20240501 14:01:02.134196 2470 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:02.134514 2470 ts_tablet_manager.cc:1367] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.005s
I20240501 14:01:02.135406 2470 raft_consensus.cc:348] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.135737 2470 raft_consensus.cc:374] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.135877 2470 raft_consensus.cc:729] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
W20240501 14:01:02.137816 32294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 109410us
I20240501 14:01:02.139010 2471 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:02.139156 2471 ts_tablet_manager.cc:1367] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.197s user 0.020s sys 0.055s
W20240501 14:01:02.139711 32295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 114668us
I20240501 14:01:02.139927 2471 raft_consensus.cc:348] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.140223 2471 raft_consensus.cc:374] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.140291 2471 raft_consensus.cc:729] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.140589 2471 consensus_queue.cc:260] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.141633 2470 consensus_queue.cc:260] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.142424 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:02.143206 29505 raft_consensus.cc:2454] T e2d4f5ec80e64aa6aceb42a0748af62b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
W20240501 14:01:02.143496 31899 leader_election.cc:341] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.143647 31899 leader_election.cc:302] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35
I20240501 14:01:02.143868 2469 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:02.144124 2469 ts_tablet_manager.cc:1367] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.201s user 0.036s sys 0.040s
I20240501 14:01:02.144276 2369 raft_consensus.cc:2726] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:02.145061 2469 raft_consensus.cc:348] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.145372 2469 raft_consensus.cc:374] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.145493 2469 raft_consensus.cc:729] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.145802 2469 consensus_queue.cc:260] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.145112 2470 ts_tablet_manager.cc:1398] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:01:02.148751 2469 ts_tablet_manager.cc:1398] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:02.151499 2471 ts_tablet_manager.cc:1398] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240501 14:01:02.155917 29101 catalog_manager.cc:5484] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:02.161531 2183 raft_consensus.cc:482] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.161747 2183 raft_consensus.cc:504] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.162487 2183 leader_election.cc:288] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:02.163200 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:02.163467 32096 raft_consensus.cc:2454] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:02.164254 32163 leader_election.cc:302] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:02.164654 2183 raft_consensus.cc:2781] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.164746 2183 raft_consensus.cc:482] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.164834 2183 raft_consensus.cc:3037] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.168046 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:01:02.168694 32163 leader_election.cc:341] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.167686 2183 raft_consensus.cc:504] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.171456 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
W20240501 14:01:02.172066 32163 leader_election.cc:341] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.173095 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:02.173303 2183 leader_election.cc:288] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:02.173364 32096 raft_consensus.cc:3037] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.173434 2465 tablet_bootstrap.cc:492] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:02.173615 2465 ts_tablet_manager.cc:1367] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.163s user 0.001s sys 0.003s
I20240501 14:01:02.174363 2465 raft_consensus.cc:348] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.174567 2465 raft_consensus.cc:374] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.174633 2465 raft_consensus.cc:729] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.174873 2465 consensus_queue.cc:260] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.175390 2465 ts_tablet_manager.cc:1398] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:02.176081 32096 raft_consensus.cc:2454] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:02.176631 32163 leader_election.cc:302] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79; no voters: 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:02.177151 2183 raft_consensus.cc:2781] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.198427 2183 raft_consensus.cc:686] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:02.198872 2183 consensus_queue.cc:237] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.235260 29097 catalog_manager.cc:5484] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:02.244602 2259 raft_consensus.cc:482] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.244824 2259 raft_consensus.cc:504] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:02.248328 2259 leader_election.cc:288] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:02.248847 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:02.249061 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0c49089dbc40efa388b025c7eb2799" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:02.249187 32096 raft_consensus.cc:2379] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9903ed35bca144bcb74e9bf5102117e7 in current term 1: Already voted for candidate 6a446ac802a7457d9f27076781f21d79 in this term.
I20240501 14:01:02.249836 31899 leader_election.cc:302] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58, 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:02.250197 2259 raft_consensus.cc:3037] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.254135 2259 raft_consensus.cc:2726] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:02.314930 2340 raft_consensus.cc:482] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.315129 2340 raft_consensus.cc:504] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.315919 2340 leader_election.cc:288] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:02.316547 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:02.316841 31965 raft_consensus.cc:2454] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:02.317260 29312 leader_election.cc:302] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:02.317276 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:02.317550 32362 raft_consensus.cc:2454] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:02.318650 2339 raft_consensus.cc:2781] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.318734 2339 raft_consensus.cc:482] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.318804 2339 raft_consensus.cc:3037] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.321336 2339 raft_consensus.cc:504] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.321892 2339 leader_election.cc:288] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:02.322441 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:02.322695 31965 raft_consensus.cc:3037] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.325307 31965 raft_consensus.cc:2454] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:02.325812 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3c16421adb477e821495e785849cc4" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:02.326066 32362 raft_consensus.cc:3037] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.329532 29312 leader_election.cc:302] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:02.329835 2339 raft_consensus.cc:2781] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.329942 2339 raft_consensus.cc:686] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:01:02.330183 2339 consensus_queue.cc:237] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.328945 32362 raft_consensus.cc:2454] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:02.336228 29097 catalog_manager.cc:5484] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:02.386132 2259 consensus_queue.cc:1035] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.398017 29098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:02.425287 2450 consensus_queue.cc:1035] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.469920 32097 raft_consensus.cc:1261] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.473858 2183 consensus_queue.cc:1035] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.498160 31965 raft_consensus.cc:1261] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.498867 2183 consensus_queue.cc:1035] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.520366 29099 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]). Expected version 1 got 0
I20240501 14:01:02.534332 29099 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]). Expected version 1 got 0
I20240501 14:01:02.540730 2562 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.541532 2562 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:02.547562 29097 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]). Expected version 1 got 0
I20240501 14:01:02.549427 2561 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.550140 2561 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:02.550791 2560 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.551533 2560 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:02.554982 29097 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f]). Expected version 1 got 0
I20240501 14:01:02.561714 29098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f] alter complete (version 1)
I20240501 14:01:02.592900 32076 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:02.595849 2340 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.598937 2340 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.623780 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:02.641978 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:02.667969 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:02.683804 2560 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.684489 2560 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:02.692957 2561 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.693482 2561 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:02.693753 2562 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.694190 2562 tablet.cc:1620] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:02.705227 29098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=f128ff1345c9458e9226809d4f01d98f] alter complete (version 2)
I20240501 14:01:02.710243 32342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:02.711936 32207 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:02.733323 29357 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:02.739163 29617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:02.743481 31943 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:02.812776 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:02.826879 2560 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.828456 2562 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.839313 29099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=f128ff1345c9458e9226809d4f01d98f] alter complete (version 3)
I20240501 14:01:02.842121 2561 tablet.cc:1722] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.850677 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:02.869386 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 8c664c3035fb47c1a937e91b38f5edcc (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.869843 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c664c3035fb47c1a937e91b38f5edcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.870878 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3876037334884a4a36bc192956cf4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.871323 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3876037334884a4a36bc192956cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.873761 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 413a79f6a61a49d9a49a5a52cec4c00f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.873765 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 8c664c3035fb47c1a937e91b38f5edcc (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.874145 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413a79f6a61a49d9a49a5a52cec4c00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.874181 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c664c3035fb47c1a937e91b38f5edcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.877912 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3876037334884a4a36bc192956cf4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.878011 2339 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.878325 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3876037334884a4a36bc192956cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.882181 32340 tablet_service.cc:1467] Processing CreateTablet for tablet 413a79f6a61a49d9a49a5a52cec4c00f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.882613 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413a79f6a61a49d9a49a5a52cec4c00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.882769 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 8c664c3035fb47c1a937e91b38f5edcc (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.883205 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c664c3035fb47c1a937e91b38f5edcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.885474 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 413a79f6a61a49d9a49a5a52cec4c00f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.886000 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413a79f6a61a49d9a49a5a52cec4c00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.889240 32074 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3876037334884a4a36bc192956cf4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.889662 32074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3876037334884a4a36bc192956cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.893193 2592 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:02.898299 2592 tablet_bootstrap.cc:654] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.919442 2592 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:02.919852 2593 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:02.920796 2592 ts_tablet_manager.cc:1367] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240501 14:01:02.921954 2592 raft_consensus.cc:348] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:02.922600 2339 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.924887 2593 tablet_bootstrap.cc:654] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.925602 2592 raft_consensus.cc:374] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.925885 2592 raft_consensus.cc:729] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.926358 2592 consensus_queue.cc:260] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:02.934125 2598 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:02.945210 29097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:02.945447 29097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:02.949496 29097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 4a0c49089dbc40efa388b025c7eb2799
I20240501 14:01:02.950523 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:02.950711 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:02.950992 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0c49089dbc40efa388b025c7eb2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:02.951288 2601 tablet_replica.cc:330] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:02.951615 2601 raft_consensus.cc:2229] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.953402 2601 raft_consensus.cc:2258] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.954875 2601 ts_tablet_manager.cc:1875] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.961472 2592 ts_tablet_manager.cc:1398] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240501 14:01:02.963075 2602 tablet_replica.cc:330] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:02.963346 2598 tablet_bootstrap.cc:654] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.963855 2600 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:02.965799 2600 tablet_bootstrap.cc:654] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.971963 2599 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:02.963347 2602 raft_consensus.cc:2229] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.973735 2602 raft_consensus.cc:2258] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.974396 2591 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:02.975193 2602 ts_tablet_manager.cc:1875] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.980825 2598 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:02.981055 2598 ts_tablet_manager.cc:1367] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240501 14:01:02.982477 2598 raft_consensus.cc:348] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.982782 2598 raft_consensus.cc:374] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.982939 2598 raft_consensus.cc:729] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.983314 2599 tablet_bootstrap.cc:654] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.983364 2598 consensus_queue.cc:260] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.984622 2603 tablet_replica.cc:330] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:02.984835 2603 raft_consensus.cc:2229] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:02.985237 2593 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:02.985229 2603 raft_consensus.cc:2258] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.985396 2593 ts_tablet_manager.cc:1367] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.008s
I20240501 14:01:02.986160 2603 ts_tablet_manager.cc:1875] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.987061 2593 raft_consensus.cc:348] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.989046 2593 raft_consensus.cc:374] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.989296 2593 raft_consensus.cc:729] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.989662 2598 ts_tablet_manager.cc:1398] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:01:02.989648 2593 consensus_queue.cc:260] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:02.991333 2591 tablet_bootstrap.cc:654] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.994217 2598 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:02.996114 2600 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:02.996405 2600 ts_tablet_manager.cc:1367] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 14:01:02.997592 2593 ts_tablet_manager.cc:1398] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:01:02.997395 2600 raft_consensus.cc:348] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:02.997967 2593 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:02.998055 2600 raft_consensus.cc:374] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.998232 2600 raft_consensus.cc:729] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.001031 2600 consensus_queue.cc:260] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.002555 2599 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:03.002810 2601 ts_tablet_manager.cc:1888] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.002833 2599 ts_tablet_manager.cc:1367] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240501 14:01:03.004297 2598 tablet_bootstrap.cc:654] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.005162 2601 log.cc:1196] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/4a0c49089dbc40efa388b025c7eb2799
I20240501 14:01:03.007463 2601 ts_tablet_manager.cc:1909] T 4a0c49089dbc40efa388b025c7eb2799 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:03.008255 2600 ts_tablet_manager.cc:1398] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:01:03.008245 2599 raft_consensus.cc:348] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.009861 2593 tablet_bootstrap.cc:654] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.010197 2599 raft_consensus.cc:374] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.010383 2599 raft_consensus.cc:729] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.010687 2599 consensus_queue.cc:260] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.013485 2599 ts_tablet_manager.cc:1398] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:01:03.014268 2021 raft_consensus.cc:482] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.014425 2021 raft_consensus.cc:504] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:03.015264 2021 leader_election.cc:288] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:03.017299 2591 tablet_bootstrap.cc:492] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:03.017443 2591 ts_tablet_manager.cc:1367] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240501 14:01:03.017743 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413a79f6a61a49d9a49a5a52cec4c00f" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:03.018169 2591 raft_consensus.cc:348] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:03.017282 2598 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:03.018335 2591 raft_consensus.cc:374] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.018388 2591 raft_consensus.cc:729] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.018543 2598 ts_tablet_manager.cc:1367] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 14:01:03.018621 2591 consensus_queue.cc:260] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:03.019268 2598 raft_consensus.cc:348] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.019459 2598 raft_consensus.cc:374] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.019567 2598 raft_consensus.cc:729] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.019541 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table_new_name [id=f128ff1345c9458e9226809d4f01d98f]) successfully deleted
I20240501 14:01:03.019848 2598 consensus_queue.cc:260] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
W20240501 14:01:03.020781 32030 leader_election.cc:341] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:03.023057 2591 ts_tablet_manager.cc:1398] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:03.026058 2598 ts_tablet_manager.cc:1398] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:03.026911 2602 ts_tablet_manager.cc:1888] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.027107 2602 log.cc:1196] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/4a0c49089dbc40efa388b025c7eb2799
I20240501 14:01:03.028523 2603 ts_tablet_manager.cc:1888] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.028694 2598 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:03.031581 2602 ts_tablet_manager.cc:1909] T 4a0c49089dbc40efa388b025c7eb2799 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:03.032727 2603 log.cc:1196] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/4a0c49089dbc40efa388b025c7eb2799
I20240501 14:01:03.032812 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table_new_name [id=f128ff1345c9458e9226809d4f01d98f]) successfully deleted
I20240501 14:01:03.033072 2598 tablet_bootstrap.cc:654] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.033460 2603 ts_tablet_manager.cc:1909] T 4a0c49089dbc40efa388b025c7eb2799 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:03.016386 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413a79f6a61a49d9a49a5a52cec4c00f" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:03.036729 32360 raft_consensus.cc:2454] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:03.037410 32031 leader_election.cc:302] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:03.037868 2213 raft_consensus.cc:2781] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.037969 2213 raft_consensus.cc:482] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.038074 2213 raft_consensus.cc:3037] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.041291 2213 raft_consensus.cc:504] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:03.042145 2213 leader_election.cc:288] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:03.042840 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413a79f6a61a49d9a49a5a52cec4c00f" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:03.043036 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413a79f6a61a49d9a49a5a52cec4c00f" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:03.043125 32360 raft_consensus.cc:3037] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.043223 29245 raft_consensus.cc:3037] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.045981 29245 raft_consensus.cc:2454] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:03.047158 32360 raft_consensus.cc:2454] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:03.048473 32030 leader_election.cc:302] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:03.049641 2213 raft_consensus.cc:2781] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.051262 2213 raft_consensus.cc:686] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:03.051600 2213 consensus_queue.cc:237] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:03.054994 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:03.066535 2598 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:03.067946 2598 ts_tablet_manager.cc:1367] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.005s
I20240501 14:01:03.069041 2598 raft_consensus.cc:348] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.070752 29617 tablet_service.cc:1467] Processing CreateTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]), partition=
I20240501 14:01:03.071139 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32c5f1cb2d471586ffb43a25f11426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.072443 31943 tablet_service.cc:1467] Processing CreateTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]), partition=
I20240501 14:01:03.072742 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32c5f1cb2d471586ffb43a25f11426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.077788 2592 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:03.070096 32077 tablet_service.cc:1467] Processing CreateTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]), partition=
I20240501 14:01:03.079244 2598 raft_consensus.cc:374] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.079371 2598 raft_consensus.cc:729] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.079396 2592 tablet_bootstrap.cc:654] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.079512 2593 tablet_bootstrap.cc:492] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:03.079557 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32c5f1cb2d471586ffb43a25f11426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.079677 2593 ts_tablet_manager.cc:1367] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.082s user 0.000s sys 0.003s
I20240501 14:01:03.079619 2598 consensus_queue.cc:260] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.080262 2598 ts_tablet_manager.cc:1398] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:01:03.080777 2593 raft_consensus.cc:348] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.081301 2593 raft_consensus.cc:374] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.081455 2593 raft_consensus.cc:729] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.081825 2593 consensus_queue.cc:260] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.084542 2593 ts_tablet_manager.cc:1398] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:03.090845 2592 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:03.091090 2592 ts_tablet_manager.cc:1367] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20240501 14:01:03.091742 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 4a0c49089dbc40efa388b025c7eb2799 (table rebalancer_dynamic_table_new_name [id=f128ff1345c9458e9226809d4f01d98f]) successfully deleted
I20240501 14:01:03.092123 2592 raft_consensus.cc:348] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.092499 2592 raft_consensus.cc:374] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.092625 2592 raft_consensus.cc:729] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.092950 2592 consensus_queue.cc:260] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.093822 2592 ts_tablet_manager.cc:1398] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:03.101423 2600 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:03.103085 2600 tablet_bootstrap.cc:654] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.110023 29098 catalog_manager.cc:5484] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:03.125501 2598 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:03.127394 2598 tablet_bootstrap.cc:654] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.137099 2600 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:03.139783 2600 ts_tablet_manager.cc:1367] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.002s
I20240501 14:01:03.140733 2600 raft_consensus.cc:348] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.141126 2600 raft_consensus.cc:374] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.141291 2600 raft_consensus.cc:729] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.141662 2600 consensus_queue.cc:260] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.143651 2600 ts_tablet_manager.cc:1398] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:03.146598 2598 tablet_bootstrap.cc:492] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:03.148283 2598 ts_tablet_manager.cc:1367] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.001s
I20240501 14:01:03.149142 2598 raft_consensus.cc:348] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.149370 2598 raft_consensus.cc:374] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.149436 2598 raft_consensus.cc:729] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.149688 2598 consensus_queue.cc:260] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.150234 2598 ts_tablet_manager.cc:1398] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:03.159534 2213 raft_consensus.cc:482] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.159732 2213 raft_consensus.cc:504] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.160462 2213 leader_election.cc:288] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:03.161345 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c664c3035fb47c1a937e91b38f5edcc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:03.161636 31965 raft_consensus.cc:2454] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:03.162487 32032 leader_election.cc:302] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:03.162767 2213 raft_consensus.cc:2781] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.162886 2213 raft_consensus.cc:482] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.162990 2213 raft_consensus.cc:3037] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.165405 32097 consensus_queue.cc:237] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:03.165787 2213 raft_consensus.cc:504] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.166379 2213 leader_election.cc:288] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:03.168092 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c664c3035fb47c1a937e91b38f5edcc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:03.168316 32361 raft_consensus.cc:3037] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.168504 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c664c3035fb47c1a937e91b38f5edcc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:03.168757 31965 raft_consensus.cc:3037] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.171152 32361 raft_consensus.cc:2454] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:03.171545 31965 raft_consensus.cc:2454] T 8c664c3035fb47c1a937e91b38f5edcc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:03.172013 32032 leader_election.cc:302] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:03.172333 2213 raft_consensus.cc:2781] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.172313 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c664c3035fb47c1a937e91b38f5edcc" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:03.172446 2213 raft_consensus.cc:686] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:03.172518 32360 raft_consensus.cc:2362] T 8c664c3035fb47c1a937e91b38f5edcc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1. Re-sending same reply.
I20240501 14:01:03.172713 2213 consensus_queue.cc:237] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.176072 29505 raft_consensus.cc:1261] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.177263 32227 raft_consensus.cc:1261] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.177727 2021 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.178470 2021 consensus_queue.cc:1035] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.194506 32228 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:03.192921 2021 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:03.205225 2219 raft_consensus.cc:482] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.205405 2219 raft_consensus.cc:504] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.206288 2219 leader_election.cc:288] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:03.209513 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3876037334884a4a36bc192956cf4" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:03.209810 32096 raft_consensus.cc:2454] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:03.213881 29572 leader_election.cc:302] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:03.214336 2219 raft_consensus.cc:2781] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.214453 2219 raft_consensus.cc:482] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.214568 2219 raft_consensus.cc:3037] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.216027 29098 catalog_manager.cc:5484] T fee3fe52d2ae462a823d3ac03f2ce8bc P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:03.217972 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3876037334884a4a36bc192956cf4" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:03.218205 32228 raft_consensus.cc:2454] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:03.221436 2219 raft_consensus.cc:504] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.222744 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3876037334884a4a36bc192956cf4" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:03.223049 32228 raft_consensus.cc:3037] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.224457 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet fee3fe52d2ae462a823d3ac03f2ce8bc with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:03.228168 32228 raft_consensus.cc:2454] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:03.228775 29571 leader_election.cc:302] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:03.229208 1937 raft_consensus.cc:2781] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.232940 2219 leader_election.cc:288] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:03.234925 2634 tablet_replica.cc:330] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:03.235175 2634 raft_consensus.cc:2229] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.235452 2634 raft_consensus.cc:2258] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.236304 2634 ts_tablet_manager.cc:1875] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:03.238380 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3876037334884a4a36bc192956cf4" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:03.239039 32095 raft_consensus.cc:3037] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.241921 32095 raft_consensus.cc:2454] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:03.242353 29098 catalog_manager.cc:5484] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:03.243544 1937 raft_consensus.cc:686] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:03.243840 1937 consensus_queue.cc:237] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.243817 29505 raft_consensus.cc:2932] T fee3fe52d2ae462a823d3ac03f2ce8bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:03.252475 29097 catalog_manager.cc:5484] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:03.258155 2634 ts_tablet_manager.cc:1888] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:03.258293 2634 log.cc:1196] T fee3fe52d2ae462a823d3ac03f2ce8bc P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/fee3fe52d2ae462a823d3ac03f2ce8bc
I20240501 14:01:03.259370 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet fee3fe52d2ae462a823d3ac03f2ce8bc (table rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]) successfully deleted
I20240501 14:01:03.284296 1937 raft_consensus.cc:482] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.284534 1937 raft_consensus.cc:504] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.285281 1937 leader_election.cc:288] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:03.285794 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32c5f1cb2d471586ffb43a25f11426" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:03.286037 32095 raft_consensus.cc:2454] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:03.286645 29572 leader_election.cc:302] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:03.287276 1937 raft_consensus.cc:2781] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.287365 1937 raft_consensus.cc:482] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.287438 1937 raft_consensus.cc:3037] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.288044 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32c5f1cb2d471586ffb43a25f11426" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:03.288257 31965 raft_consensus.cc:2454] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:03.290110 1937 raft_consensus.cc:504] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.291080 1937 leader_election.cc:288] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:03.291074 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32c5f1cb2d471586ffb43a25f11426" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:03.291297 32095 raft_consensus.cc:3037] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.291378 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32c5f1cb2d471586ffb43a25f11426" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:03.291605 31965 raft_consensus.cc:3037] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.294160 32095 raft_consensus.cc:2454] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:03.294579 29572 leader_election.cc:302] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:03.294864 1937 raft_consensus.cc:2781] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.295039 1937 raft_consensus.cc:686] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:03.295512 1937 consensus_queue.cc:237] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:03.296447 31965 raft_consensus.cc:2454] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:03.299422 29097 catalog_manager.cc:5484] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:03.447495 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:03.458068 32095 raft_consensus.cc:1261] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:03.458868 31965 raft_consensus.cc:1261] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:03.459983 2635 consensus_queue.cc:1035] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.461527 1937 consensus_queue.cc:1035] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.481680 29096 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]). Expected version 1 got 0
I20240501 14:01:03.489858 2686 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.492048 2686 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:03.493105 2687 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.496495 2687 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:03.500072 29096 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]). Expected version 1 got 0
I20240501 14:01:03.525184 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40] alter complete (version 1)
I20240501 14:01:03.532905 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:03.550117 2628 consensus_queue.cc:1035] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.550592 29100 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]). Expected version 1 got 0
I20240501 14:01:03.557051 29487 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:03.559126 32077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:03.579888 2628 consensus_queue.cc:1035] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.586021 2691 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.592226 29100 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]). Expected version 1 got 0
I20240501 14:01:03.619570 32206 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:03.621582 2691 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:03.633121 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:03.648541 2686 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:03.649111 2686 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:03.649734 2687 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:03.650205 2687 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:03.671674 29100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40] alter complete (version 2)
I20240501 14:01:03.674139 2632 consensus_queue.cc:1035] T 8c664c3035fb47c1a937e91b38f5edcc P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.676151 2632 consensus_queue.cc:1035] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.720681 29100 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table [id=d6cd9fb77d8840d6b604c17f660bbf40]). Expected version 2 got 1
I20240501 14:01:03.723459 2691 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:03.726824 2691 tablet.cc:1620] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:03.743700 31943 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:03.746378 32342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:03.751372 1937 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.785084 1937 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.787916 29357 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:03.788691 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:03.792738 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:03.802052 29617 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:03.808475 2686 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.817477 2691 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.819133 2687 tablet.cc:1722] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.824193 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=d6cd9fb77d8840d6b604c17f660bbf40] alter complete (version 3)
I20240501 14:01:03.859320 29487 tablet_service.cc:1467] Processing CreateTablet for tablet b36d195f6135436293834aec4f3296a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.859966 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36d195f6135436293834aec4f3296a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.863520 29357 tablet_service.cc:1467] Processing CreateTablet for tablet b0d8239317634874b1ffb1ba239bb9ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.863930 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d8239317634874b1ffb1ba239bb9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.874493 32342 tablet_service.cc:1467] Processing CreateTablet for tablet b36d195f6135436293834aec4f3296a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.874967 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36d195f6135436293834aec4f3296a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.877353 32340 tablet_service.cc:1467] Processing CreateTablet for tablet b0d8239317634874b1ffb1ba239bb9ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.877635 2713 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:03.877774 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d8239317634874b1ffb1ba239bb9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.879786 2713 tablet_bootstrap.cc:654] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.881908 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.882287 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:01:03.883864 29572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 43657.8us
I20240501 14:01:03.890237 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.890690 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.893220 2713 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:03.893559 2713 ts_tablet_manager.cc:1367] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 14:01:03.894681 2713 raft_consensus.cc:348] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.894994 2713 raft_consensus.cc:374] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.895095 2713 raft_consensus.cc:729] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.895380 2713 consensus_queue.cc:260] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:03.896160 2713 ts_tablet_manager.cc:1398] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:03.906155 29226 tablet_service.cc:1467] Processing CreateTablet for tablet b36d195f6135436293834aec4f3296a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.906181 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.906652 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36d195f6135436293834aec4f3296a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.909324 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.911182 2715 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:03.915462 2715 tablet_bootstrap.cc:654] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.954423 29100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:03.954686 29100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:03.958267 29617 tablet_service.cc:1467] Processing CreateTablet for tablet b0d8239317634874b1ffb1ba239bb9ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.958719 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d8239317634874b1ffb1ba239bb9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.961504 2718 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:03.965567 29100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet bd32c5f1cb2d471586ffb43a25f11426
I20240501 14:01:03.966660 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:03.967407 31943 tablet_service.cc:1514] Processing DeleteTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:03.969733 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet bd32c5f1cb2d471586ffb43a25f11426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:03.995926 2722 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:03.999313 2722 tablet_bootstrap.cc:654] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.003793 2717 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:04.003976 2723 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:04.005568 2717 tablet_bootstrap.cc:654] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.006834 2719 tablet_replica.cc:330] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:04.007123 2719 raft_consensus.cc:2229] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.007251 2721 tablet_replica.cc:330] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:04.007402 2719 raft_consensus.cc:2258] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.007658 2721 raft_consensus.cc:2229] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:04.008157 2721 raft_consensus.cc:2258] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.008386 2719 ts_tablet_manager.cc:1875] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:04.010457 2718 tablet_bootstrap.cc:654] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.011370 2720 tablet_replica.cc:330] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:04.011875 2720 raft_consensus.cc:2229] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.012217 2720 raft_consensus.cc:2258] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.014734 2721 ts_tablet_manager.cc:1875] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:04.018280 2723 tablet_bootstrap.cc:654] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.018473 2720 ts_tablet_manager.cc:1875] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:04.018360 2722 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:04.018854 2722 ts_tablet_manager.cc:1367] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240501 14:01:04.019665 2722 raft_consensus.cc:348] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.020012 2722 raft_consensus.cc:374] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.020131 2722 raft_consensus.cc:729] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.020418 2722 consensus_queue.cc:260] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.022333 2722 ts_tablet_manager.cc:1398] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:04.027863 2717 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:04.029193 2718 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:04.030704 2718 ts_tablet_manager.cc:1367] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20240501 14:01:04.031566 2718 raft_consensus.cc:348] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.031911 2718 raft_consensus.cc:374] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.032066 2718 raft_consensus.cc:729] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.032426 2718 consensus_queue.cc:260] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.033123 2717 ts_tablet_manager.cc:1367] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240501 14:01:04.033360 2718 ts_tablet_manager.cc:1398] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:04.033799 2718 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:04.037588 2717 raft_consensus.cc:348] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.039325 2718 tablet_bootstrap.cc:654] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.042562 2717 raft_consensus.cc:374] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.042719 2717 raft_consensus.cc:729] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.042814 2723 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:04.043056 2723 ts_tablet_manager.cc:1367] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240501 14:01:04.043015 2717 consensus_queue.cc:260] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.043709 2717 ts_tablet_manager.cc:1398] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:01:04.045420 2717 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:04.046953 2717 tablet_bootstrap.cc:654] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.047973 2715 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:04.048133 2715 ts_tablet_manager.cc:1367] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.137s user 0.020s sys 0.044s
I20240501 14:01:04.048928 2715 raft_consensus.cc:348] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.049304 2715 raft_consensus.cc:374] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.049463 2715 raft_consensus.cc:729] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.049810 2715 consensus_queue.cc:260] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.051074 2715 ts_tablet_manager.cc:1398] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:04.053864 2723 raft_consensus.cc:348] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.054092 2723 raft_consensus.cc:374] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.054148 2723 raft_consensus.cc:729] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.054359 2723 consensus_queue.cc:260] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.054916 2723 ts_tablet_manager.cc:1398] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:01:04.055155 2723 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:04.057669 2719 ts_tablet_manager.cc:1888] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:04.060626 2719 log.cc:1196] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/bd32c5f1cb2d471586ffb43a25f11426
I20240501 14:01:04.061744 2719 ts_tablet_manager.cc:1909] T bd32c5f1cb2d471586ffb43a25f11426 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:04.066958 2708 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.067123 2708 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.067966 2708 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:04.068549 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table_new_name [id=d6cd9fb77d8840d6b604c17f660bbf40]) successfully deleted
I20240501 14:01:04.068804 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
W20240501 14:01:04.069407 32161 leader_election.cc:341] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:04.069895 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
W20240501 14:01:04.070549 32162 leader_election.cc:341] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:04.070683 32162 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:04.071463 2708 raft_consensus.cc:2726] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:04.071892 29100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:04.077795 2721 ts_tablet_manager.cc:1888] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:04.080053 2720 ts_tablet_manager.cc:1888] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:04.080636 2723 tablet_bootstrap.cc:654] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.081377 2717 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:04.081532 2717 ts_tablet_manager.cc:1367] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240501 14:01:04.082221 2717 raft_consensus.cc:348] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.082521 2717 raft_consensus.cc:374] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.082634 2717 raft_consensus.cc:729] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.082875 2717 consensus_queue.cc:260] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.083462 2717 ts_tablet_manager.cc:1398] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:04.083971 2718 tablet_bootstrap.cc:492] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:04.084197 2718 ts_tablet_manager.cc:1367] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20240501 14:01:04.087945 2718 raft_consensus.cc:348] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.088275 2718 raft_consensus.cc:374] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.088419 2718 raft_consensus.cc:729] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.088735 2718 consensus_queue.cc:260] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.090710 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]), partition=
I20240501 14:01:04.090708 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]), partition=
I20240501 14:01:04.091085 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d89def2fcc48bcbcbc9a2a205389a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.091097 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d89def2fcc48bcbcbc9a2a205389a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.094200 2718 ts_tablet_manager.cc:1398] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:01:04.096956 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]), partition=
I20240501 14:01:04.097297 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d89def2fcc48bcbcbc9a2a205389a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.098433 2721 log.cc:1196] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd32c5f1cb2d471586ffb43a25f11426
I20240501 14:01:04.098906 2721 ts_tablet_manager.cc:1909] T bd32c5f1cb2d471586ffb43a25f11426 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:04.099968 2720 log.cc:1196] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/bd32c5f1cb2d471586ffb43a25f11426
I20240501 14:01:04.100293 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table_new_name [id=d6cd9fb77d8840d6b604c17f660bbf40]) successfully deleted
I20240501 14:01:04.102502 2720 ts_tablet_manager.cc:1909] T bd32c5f1cb2d471586ffb43a25f11426 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:04.103255 2715 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:04.103761 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet bd32c5f1cb2d471586ffb43a25f11426 (table rebalancer_dynamic_table_new_name [id=d6cd9fb77d8840d6b604c17f660bbf40]) successfully deleted
I20240501 14:01:04.104893 2715 tablet_bootstrap.cc:654] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.118590 2715 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:04.119693 2715 ts_tablet_manager.cc:1367] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240501 14:01:04.120513 2715 raft_consensus.cc:348] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.120743 2715 raft_consensus.cc:374] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.120815 2715 raft_consensus.cc:729] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.121783 2715 consensus_queue.cc:260] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.124164 2715 ts_tablet_manager.cc:1398] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:01:04.125430 2717 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:04.127197 2717 tablet_bootstrap.cc:654] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.130070 2723 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:04.130299 2723 ts_tablet_manager.cc:1367] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20240501 14:01:04.131307 2723 raft_consensus.cc:348] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.131582 2723 raft_consensus.cc:374] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.131681 2723 raft_consensus.cc:729] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.132028 2723 consensus_queue.cc:260] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.132764 2723 ts_tablet_manager.cc:1398] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:04.133697 2732 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:04.134179 2708 raft_consensus.cc:482] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.134336 2708 raft_consensus.cc:504] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.134987 2708 leader_election.cc:288] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:04.137315 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:04.136205 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:01:04.137928 32161 leader_election.cc:341] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:01:04.138084 32163 leader_election.cc:341] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:04.138192 32163 leader_election.cc:302] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:04.138820 2708 raft_consensus.cc:2726] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:04.148825 2717 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:04.149176 2717 ts_tablet_manager.cc:1367] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20240501 14:01:04.149935 2717 raft_consensus.cc:348] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.150219 2717 raft_consensus.cc:374] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.150316 2717 raft_consensus.cc:729] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.150599 2717 consensus_queue.cc:260] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.151170 2732 tablet_bootstrap.cc:654] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.155378 2717 ts_tablet_manager.cc:1398] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:01:04.170150 2732 tablet_bootstrap.cc:492] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:04.170467 2732 ts_tablet_manager.cc:1367] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240501 14:01:04.171546 2732 raft_consensus.cc:348] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.171911 2732 raft_consensus.cc:374] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.172062 2732 raft_consensus.cc:729] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.172426 2732 consensus_queue.cc:260] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.175220 2732 ts_tablet_manager.cc:1398] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:04.176774 2616 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.176942 2616 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.177788 2616 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:04.179358 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:04.179579 29245 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:01:04.179764 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:04.180008 32227 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:01:04.180035 29440 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:04.180292 2616 raft_consensus.cc:2781] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:04.180383 2616 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:04.180442 2616 raft_consensus.cc:3037] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.182384 2616 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.182893 2616 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:04.184151 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:04.184382 29245 raft_consensus.cc:3037] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.187505 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:04.187808 32227 raft_consensus.cc:3037] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.191659 32227 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:01:04.192252 29441 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:04.192646 2616 raft_consensus.cc:2781] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:04.192699 29245 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:01:04.193073 2616 raft_consensus.cc:686] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:01:04.193450 2616 consensus_queue.cc:237] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:04.194612 2340 raft_consensus.cc:482] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.194795 2340 raft_consensus.cc:504] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.195639 2340 leader_election.cc:288] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:04.196420 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d8239317634874b1ffb1ba239bb9ad" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:04.196673 29635 raft_consensus.cc:2454] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:04.197326 29310 leader_election.cc:302] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:04.198091 2339 raft_consensus.cc:2781] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:04.198180 2339 raft_consensus.cc:482] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:04.198252 2339 raft_consensus.cc:3037] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.201941 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d8239317634874b1ffb1ba239bb9ad" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:04.202226 32361 raft_consensus.cc:2454] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:04.203956 2339 raft_consensus.cc:504] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.207370 29100 catalog_manager.cc:5484] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:04.211017 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d8239317634874b1ffb1ba239bb9ad" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:04.211266 29635 raft_consensus.cc:3037] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.215272 29635 raft_consensus.cc:2454] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:04.217119 29310 leader_election.cc:302] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:04.217473 2340 raft_consensus.cc:2781] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:04.217881 2340 raft_consensus.cc:686] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:01:04.218156 2340 consensus_queue.cc:237] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:04.220528 2339 leader_election.cc:288] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:04.226459 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d8239317634874b1ffb1ba239bb9ad" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:04.226747 32362 raft_consensus.cc:3037] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.229043 32362 raft_consensus.cc:2454] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:04.230651 2747 raft_consensus.cc:482] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.230885 2747 raft_consensus.cc:504] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.232297 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36d195f6135436293834aec4f3296a2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:04.232534 29505 raft_consensus.cc:2454] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:04.233103 32295 leader_election.cc:302] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:04.233564 2699 raft_consensus.cc:2781] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:04.233667 2699 raft_consensus.cc:482] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:04.233798 2699 raft_consensus.cc:3037] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.234436 2747 leader_election.cc:288] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:04.234853 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36d195f6135436293834aec4f3296a2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:04.235069 29245 raft_consensus.cc:2454] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:04.237519 2699 raft_consensus.cc:504] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.239127 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36d195f6135436293834aec4f3296a2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:04.239308 29505 raft_consensus.cc:3037] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.239799 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36d195f6135436293834aec4f3296a2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:04.240031 29245 raft_consensus.cc:3037] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.240041 29100 catalog_manager.cc:5484] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:04.242004 29505 raft_consensus.cc:2454] T b36d195f6135436293834aec4f3296a2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:04.242607 32295 leader_election.cc:302] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:04.243049 2747 raft_consensus.cc:2781] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:04.245343 29245 raft_consensus.cc:2454] T b36d195f6135436293834aec4f3296a2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:04.247081 2699 leader_election.cc:288] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:04.249620 2747 raft_consensus.cc:686] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:04.249975 2747 consensus_queue.cc:237] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.253834 29100 catalog_manager.cc:5484] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:04.266376 29245 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:04.268433 29377 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:04.268487 32362 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:04.268941 1939 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:04.269254 2289 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:04.272383 1939 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:04.273415 29377 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:04.276966 32362 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:04.279238 29100 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:04.285023 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 578c3f4c5a7c4e2c962b64736195f0bc with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:04.285708 2763 tablet_replica.cc:330] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:04.286265 2763 raft_consensus.cc:2229] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.286554 2763 raft_consensus.cc:2258] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.287195 2763 ts_tablet_manager.cc:1875] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.293184 2763 ts_tablet_manager.cc:1888] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:04.293314 2763 log.cc:1196] T 578c3f4c5a7c4e2c962b64736195f0bc P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/578c3f4c5a7c4e2c962b64736195f0bc
I20240501 14:01:04.294152 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 578c3f4c5a7c4e2c962b64736195f0bc (table rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]) successfully deleted
I20240501 14:01:04.326805 1939 raft_consensus.cc:482] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:04.326987 1939 raft_consensus.cc:504] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.327831 1939 leader_election.cc:288] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:04.328256 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:04.328487 32227 raft_consensus.cc:2454] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:04.328859 29180 leader_election.cc:302] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:04.329032 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:04.329142 1939 raft_consensus.cc:2781] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:04.329200 1939 raft_consensus.cc:482] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:04.329239 1939 raft_consensus.cc:3037] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.329244 32095 raft_consensus.cc:2454] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:04.331624 1939 raft_consensus.cc:504] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.332180 1939 leader_election.cc:288] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:04.332574 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:04.332688 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d89def2fcc48bcbcbc9a2a205389a1" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:04.332791 32227 raft_consensus.cc:3037] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.332914 32095 raft_consensus.cc:3037] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:04.335093 32227 raft_consensus.cc:2454] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:04.335464 29180 leader_election.cc:302] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:04.335701 1939 raft_consensus.cc:2781] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:04.335690 32095 raft_consensus.cc:2454] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:04.335829 1939 raft_consensus.cc:686] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:04.336652 1939 consensus_queue.cc:237] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:04.340440 29096 catalog_manager.cc:5484] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:04.468022 29102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:04.514117 32095 raft_consensus.cc:1261] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:04.516384 2762 consensus_queue.cc:1035] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.523473 32228 raft_consensus.cc:1261] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:04.524539 2762 consensus_queue.cc:1035] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.547312 29102 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]). Expected version 1 got 0
I20240501 14:01:04.573216 2813 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:04.573836 2813 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:04.574381 2814 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:04.575109 2814 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:04.576292 29102 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]). Expected version 1 got 0
I20240501 14:01:04.578977 2818 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:04.579598 2818 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:04.582353 29099 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]). Expected version 1 got 0
I20240501 14:01:04.592839 29102 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820]). Expected version 1 got 0
I20240501 14:01:04.618777 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820] alter complete (version 1)
I20240501 14:01:04.680166 2616 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.705133 29487 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:04.707523 32077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:04.709455 29226 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:04.721138 31943 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:04.727443 32206 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:04.737146 2311 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.752743 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:04.764601 2813 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.765169 2813 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:04.774127 2814 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.774704 2814 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:04.776086 29357 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:01:04.785449 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:04.789147 2747 consensus_queue.cc:1035] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.790913 2818 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.791448 2818 tablet.cc:1620] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:04.799274 29616 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c1d804ba74db794bc0abe68bc9f96 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.799721 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c1d804ba74db794bc0abe68bc9f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.799638 32077 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c1d804ba74db794bc0abe68bc9f96 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.800083 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c1d804ba74db794bc0abe68bc9f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.806753 2843 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:04.809082 2843 tablet_bootstrap.cc:654] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.809502 29617 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:04.811403 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=3144a925db1e41f89cc8bd06696d4820] alter complete (version 2)
I20240501 14:01:04.813913 31943 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c1d804ba74db794bc0abe68bc9f96 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.814411 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c1d804ba74db794bc0abe68bc9f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.814674 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 66a151a1d8f74a5897dd6c8e4b2a7057 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.815263 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a151a1d8f74a5897dd6c8e4b2a7057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.812956 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 66a151a1d8f74a5897dd6c8e4b2a7057 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.817540 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a151a1d8f74a5897dd6c8e4b2a7057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.820660 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 66a151a1d8f74a5897dd6c8e4b2a7057 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.821137 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a151a1d8f74a5897dd6c8e4b2a7057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.821162 31944 tablet_service.cc:1467] Processing CreateTablet for tablet e6f045ec0d8448dc9fed9f14a7990170 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.821678 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f045ec0d8448dc9fed9f14a7990170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.826282 29357 tablet_service.cc:1467] Processing CreateTablet for tablet e6f045ec0d8448dc9fed9f14a7990170 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.826718 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f045ec0d8448dc9fed9f14a7990170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.831791 29487 tablet_service.cc:1467] Processing CreateTablet for tablet e6f045ec0d8448dc9fed9f14a7990170 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:04.832199 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f045ec0d8448dc9fed9f14a7990170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.845211 2340 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.889230 2855 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:04.905015 2855 tablet_bootstrap.cc:654] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.914470 29104 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:04.916405 32340 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:01:04.919396 29571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 62546.3us
I20240501 14:01:04.924188 2813 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:04.925817 2818 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:04.927240 2855 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:04.927417 2855 ts_tablet_manager.cc:1367] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.004s
I20240501 14:01:04.938864 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=3144a925db1e41f89cc8bd06696d4820] alter complete (version 3)
I20240501 14:01:04.941709 2843 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:04.941869 2843 ts_tablet_manager.cc:1367] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.034s
I20240501 14:01:04.941380 2855 raft_consensus.cc:348] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:04.941963 2855 raft_consensus.cc:374] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.942077 2855 raft_consensus.cc:729] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.942848 2855 consensus_queue.cc:260] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:04.943871 2843 raft_consensus.cc:348] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:04.944061 2843 raft_consensus.cc:374] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:04.944130 2843 raft_consensus.cc:729] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:04.944331 2843 consensus_queue.cc:260] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:04.948894 2855 ts_tablet_manager.cc:1398] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.021s user 0.000s sys 0.003s
I20240501 14:01:04.949368 2843 ts_tablet_manager.cc:1398] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:04.951752 2814 tablet.cc:1722] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:04.961933 2340 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.970430 2840 consensus_queue.cc:1035] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.987769 2847 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:04.989678 2847 tablet_bootstrap.cc:654] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.991067 2848 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:04.992835 2848 tablet_bootstrap.cc:654] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.994648 2852 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:04.996695 2852 tablet_bootstrap.cc:654] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:04.998054 2853 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:04.999675 2853 tablet_bootstrap.cc:654] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.034951 2845 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:05.037499 2845 tablet_bootstrap.cc:654] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.046749 29104 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:05.047125 29104 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:05.052142 29104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 64d89def2fcc48bcbcbc9a2a205389a1
I20240501 14:01:05.052858 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:05.060642 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:05.063890 2867 tablet_replica.cc:330] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:05.070899 2867 raft_consensus.cc:2229] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:05.071352 2867 raft_consensus.cc:2258] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.072276 2867 ts_tablet_manager.cc:1875] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.074779 2853 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:05.079497 2847 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:05.079803 2853 ts_tablet_manager.cc:1367] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20240501 14:01:05.080953 2852 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:05.082134 2853 raft_consensus.cc:348] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.065408 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d89def2fcc48bcbcbc9a2a205389a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:05.097931 2847 ts_tablet_manager.cc:1367] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.004s
I20240501 14:01:05.098848 2847 raft_consensus.cc:348] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.099141 2847 raft_consensus.cc:374] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.099242 2847 raft_consensus.cc:729] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.099490 2847 consensus_queue.cc:260] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.100394 2847 ts_tablet_manager.cc:1398] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:05.101773 2852 ts_tablet_manager.cc:1367] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.002s
I20240501 14:01:05.102696 2852 raft_consensus.cc:348] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.103056 2852 raft_consensus.cc:374] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.103191 2852 raft_consensus.cc:729] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.103488 2852 consensus_queue.cc:260] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.105525 2870 tablet_replica.cc:330] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:05.105836 2870 raft_consensus.cc:2229] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.106186 2870 raft_consensus.cc:2258] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.109458 2853 raft_consensus.cc:374] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.109558 2853 raft_consensus.cc:729] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.109834 2853 consensus_queue.cc:260] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.110419 2853 ts_tablet_manager.cc:1398] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
I20240501 14:01:05.111227 2870 ts_tablet_manager.cc:1875] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.111466 2866 tablet_replica.cc:330] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:05.111765 2866 raft_consensus.cc:2229] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.112043 2866 raft_consensus.cc:2258] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.113073 2866 ts_tablet_manager.cc:1875] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.121938 2848 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:05.122185 2848 ts_tablet_manager.cc:1367] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.000s
I20240501 14:01:05.123256 2848 raft_consensus.cc:348] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.116448 2845 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:05.129177 2845 ts_tablet_manager.cc:1367] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.000s
I20240501 14:01:05.130129 2845 raft_consensus.cc:348] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.130404 2845 raft_consensus.cc:374] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.130597 2845 raft_consensus.cc:729] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.133476 2852 ts_tablet_manager.cc:1398] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240501 14:01:05.137269 2870 ts_tablet_manager.cc:1888] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:05.142853 2840 raft_consensus.cc:482] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:05.143036 2840 raft_consensus.cc:504] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.143925 2840 leader_election.cc:288] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:05.145253 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:05.145488 32227 raft_consensus.cc:2454] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:05.146111 32295 leader_election.cc:302] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:05.146396 2840 raft_consensus.cc:2781] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:05.146495 2840 raft_consensus.cc:482] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:05.146602 2840 raft_consensus.cc:3037] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.151355 2840 raft_consensus.cc:504] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.144500 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:01:05.152143 32296 leader_election.cc:341] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:05.152284 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:05.152513 2840 leader_election.cc:288] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
W20240501 14:01:05.152865 32296 leader_election.cc:341] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:05.153120 2870 log.cc:1196] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/64d89def2fcc48bcbcbc9a2a205389a1
I20240501 14:01:05.153484 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:05.153697 32228 raft_consensus.cc:3037] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.154176 2870 ts_tablet_manager.cc:1909] T 64d89def2fcc48bcbcbc9a2a205389a1 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:05.123559 2848 raft_consensus.cc:374] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.156407 2848 raft_consensus.cc:729] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.156455 32228 raft_consensus.cc:2454] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:05.153069 2845 consensus_queue.cc:260] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.157397 32295 leader_election.cc:302] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:05.157462 2845 ts_tablet_manager.cc:1398] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240501 14:01:05.157719 2845 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:05.157784 2840 raft_consensus.cc:2781] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:05.157949 2840 raft_consensus.cc:686] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:05.158190 2840 consensus_queue.cc:237] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.159377 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table_new_name [id=3144a925db1e41f89cc8bd06696d4820]) successfully deleted
I20240501 14:01:05.162227 2845 tablet_bootstrap.cc:654] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.167080 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:05.172952 2845 tablet_bootstrap.cc:492] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:05.173200 2845 ts_tablet_manager.cc:1367] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240501 14:01:05.172722 2848 consensus_queue.cc:260] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.174582 2845 raft_consensus.cc:348] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.174834 2845 raft_consensus.cc:374] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.174898 2845 raft_consensus.cc:729] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.175140 2845 consensus_queue.cc:260] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.179579 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]), partition=
I20240501 14:01:05.179951 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8f28cc84544562b8d7c25f59452a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:05.180019 2845 ts_tablet_manager.cc:1398] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 14:01:05.180065 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]), partition=
I20240501 14:01:05.180400 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8f28cc84544562b8d7c25f59452a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:05.185153 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]), partition=
I20240501 14:01:05.185530 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8f28cc84544562b8d7c25f59452a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:05.186676 2855 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:05.188163 2855 tablet_bootstrap.cc:654] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.193367 2847 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:05.196673 2847 tablet_bootstrap.cc:654] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.201694 2848 ts_tablet_manager.cc:1398] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20240501 14:01:05.202618 2848 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:05.203014 2847 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:05.203249 2847 ts_tablet_manager.cc:1367] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 14:01:05.202549 2855 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:05.203740 2855 ts_tablet_manager.cc:1367] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240501 14:01:05.204675 2848 tablet_bootstrap.cc:654] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.204864 2847 raft_consensus.cc:348] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.205097 2847 raft_consensus.cc:374] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.205193 2847 raft_consensus.cc:729] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.205415 2847 consensus_queue.cc:260] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.205770 2855 raft_consensus.cc:348] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.205983 2855 raft_consensus.cc:374] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.206049 2855 raft_consensus.cc:729] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.206290 2855 consensus_queue.cc:260] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.206957 2855 ts_tablet_manager.cc:1398] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:01:05.205973 2847 ts_tablet_manager.cc:1398] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:05.217046 2867 ts_tablet_manager.cc:1888] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:05.217228 2867 log.cc:1196] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/64d89def2fcc48bcbcbc9a2a205389a1
I20240501 14:01:05.217651 2867 ts_tablet_manager.cc:1909] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:01:05.218786 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table_new_name [id=3144a925db1e41f89cc8bd06696d4820]) successfully deleted
I20240501 14:01:05.219314 2866 ts_tablet_manager.cc:1888] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:05.219457 2866 log.cc:1196] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/64d89def2fcc48bcbcbc9a2a205389a1
I20240501 14:01:05.219947 2866 ts_tablet_manager.cc:1909] T 64d89def2fcc48bcbcbc9a2a205389a1 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:05.220932 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 64d89def2fcc48bcbcbc9a2a205389a1 (table rebalancer_dynamic_table_new_name [id=3144a925db1e41f89cc8bd06696d4820]) successfully deleted
I20240501 14:01:05.226042 2848 tablet_bootstrap.cc:492] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:05.226303 2848 ts_tablet_manager.cc:1367] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20240501 14:01:05.227174 2848 raft_consensus.cc:348] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.227381 2848 raft_consensus.cc:374] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.227460 2848 raft_consensus.cc:729] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.227707 2848 consensus_queue.cc:260] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.228289 2848 ts_tablet_manager.cc:1398] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:05.228816 2848 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:05.230376 2848 tablet_bootstrap.cc:654] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:05.235802 29102 catalog_manager.cc:5484] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:05.239908 2848 tablet_bootstrap.cc:492] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:05.240077 2848 ts_tablet_manager.cc:1367] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:01:05.240872 2848 raft_consensus.cc:348] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.244366 2848 raft_consensus.cc:374] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:05.244555 2848 raft_consensus.cc:729] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:05.245879 2848 consensus_queue.cc:260] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.246719 2848 ts_tablet_manager.cc:1398] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:01:05.273880 2340 raft_consensus.cc:482] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:05.274061 2340 raft_consensus.cc:504] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.274966 2340 leader_election.cc:288] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:05.275581 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8f28cc84544562b8d7c25f59452a49" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:05.275638 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8f28cc84544562b8d7c25f59452a49" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:05.275852 31965 raft_consensus.cc:2454] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:05.275866 32227 raft_consensus.cc:2454] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:05.276284 29311 leader_election.cc:302] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:05.276736 2340 raft_consensus.cc:2781] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:05.276835 2340 raft_consensus.cc:482] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:05.276929 2340 raft_consensus.cc:3037] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.278976 2340 raft_consensus.cc:504] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.279668 2340 leader_election.cc:288] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:05.280100 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8f28cc84544562b8d7c25f59452a49" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:05.280174 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8f28cc84544562b8d7c25f59452a49" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:05.280301 32227 raft_consensus.cc:3037] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.280409 31965 raft_consensus.cc:3037] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.283262 32227 raft_consensus.cc:2454] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:05.283659 29311 leader_election.cc:302] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:05.283679 31965 raft_consensus.cc:2454] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:05.283962 2340 raft_consensus.cc:2781] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:05.284265 2340 raft_consensus.cc:686] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:01:05.284516 2340 consensus_queue.cc:237] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.289011 29102 catalog_manager.cc:5484] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:05.319118 31965 consensus_queue.cc:237] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:05.322427 29377 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:05.322762 29244 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:05.323226 2450 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:05.323895 2896 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:05.327685 2896 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:05.328538 29377 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:05.334913 29104 catalog_manager.cc:5484] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:05.337642 29244 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:05.339363 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c86d880c077411a99e6b9597adeeb0d with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:05.339757 2870 tablet_replica.cc:330] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:05.340003 2870 raft_consensus.cc:2229] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.340193 2870 raft_consensus.cc:2258] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.340831 2870 ts_tablet_manager.cc:1875] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:05.341331 2856 raft_consensus.cc:482] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:05.341459 2856 raft_consensus.cc:504] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.342099 2856 leader_election.cc:288] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:05.342523 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c1d804ba74db794bc0abe68bc9f96" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:05.342787 32097 raft_consensus.cc:2454] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:05.342756 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c1d804ba74db794bc0abe68bc9f96" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:05.342978 31965 raft_consensus.cc:2454] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:05.343313 29572 leader_election.cc:302] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:05.343823 2856 raft_consensus.cc:2781] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:05.343907 2856 raft_consensus.cc:482] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:05.343962 2856 raft_consensus.cc:3037] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.346751 2856 raft_consensus.cc:504] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.348018 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c1d804ba74db794bc0abe68bc9f96" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:05.348028 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c1d804ba74db794bc0abe68bc9f96" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:05.348239 32096 raft_consensus.cc:3037] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.348253 31965 raft_consensus.cc:3037] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.349184 2870 ts_tablet_manager.cc:1888] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:05.349318 2870 log.cc:1196] T 6c86d880c077411a99e6b9597adeeb0d P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c86d880c077411a99e6b9597adeeb0d
I20240501 14:01:05.350174 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 6c86d880c077411a99e6b9597adeeb0d (table rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]) successfully deleted
I20240501 14:01:05.350827 31965 raft_consensus.cc:2454] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:05.351063 2856 leader_election.cc:288] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:05.351404 32096 raft_consensus.cc:2454] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:05.351871 29572 leader_election.cc:302] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:05.352131 2856 raft_consensus.cc:2781] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:05.352615 2856 raft_consensus.cc:686] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:05.352865 2856 consensus_queue.cc:237] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:05.360236 29096 catalog_manager.cc:5484] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:05.377938 2896 raft_consensus.cc:482] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:05.378118 2896 raft_consensus.cc:504] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.378829 2896 leader_election.cc:288] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:05.379258 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f045ec0d8448dc9fed9f14a7990170" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:05.379285 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f045ec0d8448dc9fed9f14a7990170" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:05.379484 29505 raft_consensus.cc:2454] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:05.379523 29377 raft_consensus.cc:2454] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:05.379932 31899 leader_election.cc:302] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:05.380236 2896 raft_consensus.cc:2781] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:05.380347 2896 raft_consensus.cc:482] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:05.380437 2896 raft_consensus.cc:3037] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.382772 2896 raft_consensus.cc:504] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.383563 2896 leader_election.cc:288] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:05.383757 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f045ec0d8448dc9fed9f14a7990170" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:05.383848 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f045ec0d8448dc9fed9f14a7990170" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:05.383949 29377 raft_consensus.cc:3037] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.384019 29505 raft_consensus.cc:3037] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.385957 29377 raft_consensus.cc:2454] T e6f045ec0d8448dc9fed9f14a7990170 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:05.386318 31900 leader_election.cc:302] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:05.386574 2896 raft_consensus.cc:2781] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:05.386636 29505 raft_consensus.cc:2454] T e6f045ec0d8448dc9fed9f14a7990170 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:05.386786 2896 raft_consensus.cc:686] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:05.387126 2896 consensus_queue.cc:237] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:05.391163 29096 catalog_manager.cc:5484] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:05.397367 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:05.403889 32228 raft_consensus.cc:1261] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:05.404428 2340 consensus_queue.cc:1035] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.404698 31963 raft_consensus.cc:1261] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:05.405822 2340 consensus_queue.cc:1035] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.409253 29096 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]). Expected version 1 got 0
I20240501 14:01:05.416582 2818 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:05.417217 2818 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:05.419564 2919 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:05.420424 2919 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:05.421586 29104 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]). Expected version 1 got 0
I20240501 14:01:05.423959 29104 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b]). Expected version 1 got 0
I20240501 14:01:05.431686 2917 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:05.432412 2917 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:01:05.435014 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b] alter complete (version 1)
I20240501 14:01:05.529431 2632 raft_consensus.cc:482] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:05.529680 2632 raft_consensus.cc:504] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:05.530692 2632 leader_election.cc:288] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:05.531328 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:05.531914 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a151a1d8f74a5897dd6c8e4b2a7057" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:05.532182 32228 raft_consensus.cc:2379] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a697f3a1ee544daa7073814ab845d58 in current term 1: Already voted for candidate 6edd6a3430d44c38a19c2804e555059f in this term.
I20240501 14:01:05.533008 32031 leader_election.cc:302] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:05.533488 2632 raft_consensus.cc:3037] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:05.537889 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:05.547901 2632 raft_consensus.cc:2726] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:05.549374 2840 consensus_queue.cc:1035] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.560923 2850 consensus_queue.cc:1035] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.571134 2917 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:05.571617 2917 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:01:05.572643 2818 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:05.573308 2818 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:05.582408 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=32f0b98be1494aeb9c7b115f15321f7b] alter complete (version 2)
I20240501 14:01:05.585919 2919 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:05.586628 2919 tablet.cc:1620] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:05.703088 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:05.714617 32077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:05.724715 2917 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:05.732606 2919 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:05.739138 2818 tablet.cc:1722] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:05.752511 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=32f0b98be1494aeb9c7b115f15321f7b] alter complete (version 3)
I20240501 14:01:05.770460 29226 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.800714 31943 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:05.803359 29616 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:05.822420 2856 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.827729 32342 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:05.830983 29357 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.834667 29487 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:05.848568 2856 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.856206 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:05.856478 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:05.861375 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 3a8f28cc84544562b8d7c25f59452a49
I20240501 14:01:05.862793 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:05.863773 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:01:05.871417 31943 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8f28cc84544562b8d7c25f59452a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:05.916473 2978 tablet_replica.cc:330] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:01:05.916728 2978 raft_consensus.cc:2229] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:05.917382 2978 raft_consensus.cc:2258] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.918422 2978 ts_tablet_manager.cc:1875] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.921521 2979 tablet_replica.cc:330] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:05.921882 2979 raft_consensus.cc:2229] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.922165 2979 raft_consensus.cc:2258] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.922591 2977 tablet_replica.cc:330] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:05.924935 2979 ts_tablet_manager.cc:1875] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.924854 32207 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:05.925320 2977 raft_consensus.cc:2229] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.925549 2977 raft_consensus.cc:2258] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.929189 2977 ts_tablet_manager.cc:1875] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:05.932632 2897 consensus_queue.cc:1035] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.961177 2977 ts_tablet_manager.cc:1888] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:05.961475 2977 log.cc:1196] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/3a8f28cc84544562b8d7c25f59452a49
I20240501 14:01:05.962070 2977 ts_tablet_manager.cc:1909] T 3a8f28cc84544562b8d7c25f59452a49 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:05.963311 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table_new_name [id=32f0b98be1494aeb9c7b115f15321f7b]) successfully deleted
I20240501 14:01:05.978235 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:05.980299 2979 ts_tablet_manager.cc:1888] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:05.980453 2979 log.cc:1196] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/3a8f28cc84544562b8d7c25f59452a49
I20240501 14:01:05.990622 29487 tablet_service.cc:1467] Processing CreateTablet for tablet a06abcab91b2430989e8d37175bc239d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]), partition=
I20240501 14:01:05.990996 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06abcab91b2430989e8d37175bc239d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:05.995208 29226 tablet_service.cc:1467] Processing CreateTablet for tablet a06abcab91b2430989e8d37175bc239d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]), partition=
I20240501 14:01:05.995633 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06abcab91b2430989e8d37175bc239d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:05.999914 29616 tablet_service.cc:1467] Processing CreateTablet for tablet a06abcab91b2430989e8d37175bc239d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]), partition=
I20240501 14:01:06.000298 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06abcab91b2430989e8d37175bc239d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.003176 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:06.010042 2978 ts_tablet_manager.cc:1888] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:06.012940 2978 log.cc:1196] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a8f28cc84544562b8d7c25f59452a49
I20240501 14:01:06.013773 2978 ts_tablet_manager.cc:1909] T 3a8f28cc84544562b8d7c25f59452a49 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:01:05.995931 2450 consensus_queue.cc:1035] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20240501 14:01:06.019032 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table_new_name [id=32f0b98be1494aeb9c7b115f15321f7b]) successfully deleted
I20240501 14:01:05.980947 2979 ts_tablet_manager.cc:1909] T 3a8f28cc84544562b8d7c25f59452a49 P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:06.035089 2983 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:06.037217 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 9342fb2bff81431db52967c7bf4d331a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.037494 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 9342fb2bff81431db52967c7bf4d331a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.037676 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9342fb2bff81431db52967c7bf4d331a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.037897 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9342fb2bff81431db52967c7bf4d331a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.041424 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 9342fb2bff81431db52967c7bf4d331a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.041837 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9342fb2bff81431db52967c7bf4d331a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.042114 2984 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:06.047396 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7c0ef199394de490b790a26532df3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.047653 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7c0ef199394de490b790a26532df3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.047888 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7c0ef199394de490b790a26532df3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.048061 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7c0ef199394de490b790a26532df3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.048528 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 3a8f28cc84544562b8d7c25f59452a49 (table rebalancer_dynamic_table_new_name [id=32f0b98be1494aeb9c7b115f15321f7b]) successfully deleted
I20240501 14:01:06.061708 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 5351f83631e64b7abe60575a6c285592 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.061921 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 5351f83631e64b7abe60575a6c285592 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.062179 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 5351f83631e64b7abe60575a6c285592 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.062330 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351f83631e64b7abe60575a6c285592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.062456 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351f83631e64b7abe60575a6c285592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.062665 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351f83631e64b7abe60575a6c285592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.068082 2987 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:06.069757 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7c0ef199394de490b790a26532df3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.070176 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7c0ef199394de490b790a26532df3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.074903 2987 tablet_bootstrap.cc:654] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.075886 2988 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:06.082082 2983 tablet_bootstrap.cc:654] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.085160 2989 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:06.087103 2991 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:06.087779 2981 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:06.090431 2989 tablet_bootstrap.cc:654] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.052258 2984 tablet_bootstrap.cc:654] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.092216 2991 tablet_bootstrap.cc:654] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.093191 2988 tablet_bootstrap.cc:654] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.096395 2981 tablet_bootstrap.cc:654] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.098629 2987 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:06.098860 2987 ts_tablet_manager.cc:1367] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240501 14:01:06.099705 2987 raft_consensus.cc:348] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.100049 2987 raft_consensus.cc:374] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.100189 2987 raft_consensus.cc:729] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.100531 2987 consensus_queue.cc:260] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.102949 2987 ts_tablet_manager.cc:1398] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:06.109221 2989 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:06.109467 2989 ts_tablet_manager.cc:1367] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240501 14:01:06.110633 2989 raft_consensus.cc:348] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.110913 2989 raft_consensus.cc:374] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.111028 2989 raft_consensus.cc:729] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.111325 2989 consensus_queue.cc:260] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.112025 2988 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:06.112268 2988 ts_tablet_manager.cc:1367] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240501 14:01:06.115856 2989 ts_tablet_manager.cc:1398] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:06.121115 2991 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:06.121343 2991 ts_tablet_manager.cc:1367] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20240501 14:01:06.122225 2991 raft_consensus.cc:348] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.122465 2984 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:06.122486 2991 raft_consensus.cc:374] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.122588 2991 raft_consensus.cc:729] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.122689 2984 ts_tablet_manager.cc:1367] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.000s
I20240501 14:01:06.122880 2991 consensus_queue.cc:260] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.122997 2988 raft_consensus.cc:348] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.124079 2991 ts_tablet_manager.cc:1398] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:06.124567 2988 raft_consensus.cc:374] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.124723 2988 raft_consensus.cc:729] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.125020 2988 consensus_queue.cc:260] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.126191 2988 ts_tablet_manager.cc:1398] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:01:06.142052 2981 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:06.142253 2981 ts_tablet_manager.cc:1367] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.000s
I20240501 14:01:06.143142 2981 raft_consensus.cc:348] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.143396 2981 raft_consensus.cc:374] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.143460 2981 raft_consensus.cc:729] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.143661 2981 consensus_queue.cc:260] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.144196 2981 ts_tablet_manager.cc:1398] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:06.147392 2981 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:06.150839 2984 raft_consensus.cc:348] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.151453 2984 raft_consensus.cc:374] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.151544 2984 raft_consensus.cc:729] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.153856 2984 consensus_queue.cc:260] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.157802 2984 ts_tablet_manager.cc:1398] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:01:06.161969 2981 tablet_bootstrap.cc:654] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.161980 2984 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:06.163411 2984 tablet_bootstrap.cc:654] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.165036 2983 tablet_bootstrap.cc:492] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:06.165683 2983 ts_tablet_manager.cc:1367] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.003s
I20240501 14:01:06.166348 2983 raft_consensus.cc:348] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.166548 2983 raft_consensus.cc:374] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.166653 2983 raft_consensus.cc:729] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.166913 2983 consensus_queue.cc:260] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.167516 2983 ts_tablet_manager.cc:1398] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:06.169562 2983 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:06.173048 2983 tablet_bootstrap.cc:654] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.179107 2983 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:06.179287 2983 ts_tablet_manager.cc:1367] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 14:01:06.185954 2983 raft_consensus.cc:348] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.186214 2983 raft_consensus.cc:374] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.186304 2983 raft_consensus.cc:729] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.186544 2983 consensus_queue.cc:260] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.188364 2983 ts_tablet_manager.cc:1398] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240501 14:01:06.202102 2984 tablet_bootstrap.cc:492] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:06.202292 2984 ts_tablet_manager.cc:1367] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240501 14:01:06.203887 2984 raft_consensus.cc:348] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.204094 2984 raft_consensus.cc:374] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.204147 2984 raft_consensus.cc:729] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.204419 2984 consensus_queue.cc:260] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.205098 2984 ts_tablet_manager.cc:1398] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:06.205716 2984 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:06.213613 2984 tablet_bootstrap.cc:654] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.215582 2984 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:06.215669 2981 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:06.215719 2984 ts_tablet_manager.cc:1367] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:01:06.215852 2981 ts_tablet_manager.cc:1367] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.000s
I20240501 14:01:06.216392 2984 raft_consensus.cc:348] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.216589 2984 raft_consensus.cc:374] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.216647 2984 raft_consensus.cc:729] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.216817 2981 raft_consensus.cc:348] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.217219 2981 raft_consensus.cc:374] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.217284 2981 raft_consensus.cc:729] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.217489 2981 consensus_queue.cc:260] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.218060 2981 ts_tablet_manager.cc:1398] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:06.216910 2984 consensus_queue.cc:260] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.222321 2984 ts_tablet_manager.cc:1398] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:06.223038 2984 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:06.224469 2984 tablet_bootstrap.cc:654] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.233911 2984 tablet_bootstrap.cc:492] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:06.234056 2984 ts_tablet_manager.cc:1367] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240501 14:01:06.234555 2984 raft_consensus.cc:348] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.234714 2984 raft_consensus.cc:374] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:06.234771 2984 raft_consensus.cc:729] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:06.234974 2984 consensus_queue.cc:260] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.236128 2984 ts_tablet_manager.cc:1398] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:06.255563 1937 raft_consensus.cc:482] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:06.255712 1937 raft_consensus.cc:504] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.256494 1937 leader_election.cc:288] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:06.257068 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351f83631e64b7abe60575a6c285592" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:06.257288 29507 raft_consensus.cc:2454] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:06.257330 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351f83631e64b7abe60575a6c285592" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:06.257591 32228 raft_consensus.cc:2454] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:06.258001 29571 leader_election.cc:302] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:06.258456 1937 raft_consensus.cc:2781] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:06.258513 1937 raft_consensus.cc:482] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:06.258561 1937 raft_consensus.cc:3037] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.260895 1937 raft_consensus.cc:504] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.261714 1937 leader_election.cc:288] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:06.262195 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351f83631e64b7abe60575a6c285592" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:06.262450 32228 raft_consensus.cc:3037] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.263337 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351f83631e64b7abe60575a6c285592" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:06.263536 29507 raft_consensus.cc:3037] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.265308 32228 raft_consensus.cc:2454] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:06.265731 29571 leader_election.cc:302] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:06.266212 29507 raft_consensus.cc:2454] T 5351f83631e64b7abe60575a6c285592 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:06.267055 1937 raft_consensus.cc:2781] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:06.267297 1937 raft_consensus.cc:686] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:06.267963 1937 consensus_queue.cc:237] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:06.270159 29506 consensus_queue.cc:237] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.273630 29096 catalog_manager.cc:5484] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:06.275194 31963 raft_consensus.cc:1261] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:06.275339 29244 raft_consensus.cc:1261] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:06.275952 2616 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:06.276309 2964 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:06.281489 2213 raft_consensus.cc:482] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:06.281669 2213 raft_consensus.cc:504] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.282459 2213 leader_election.cc:288] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:06.282996 2964 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:06.283290 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9342fb2bff81431db52967c7bf4d331a" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:06.283509 29245 raft_consensus.cc:2454] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:06.284719 32030 leader_election.cc:302] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:06.285765 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9342fb2bff81431db52967c7bf4d331a" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:06.286005 29637 raft_consensus.cc:2454] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:06.287386 2213 raft_consensus.cc:2781] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:06.287482 2213 raft_consensus.cc:482] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:06.287562 2213 raft_consensus.cc:3037] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.289443 2213 raft_consensus.cc:504] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.289618 29096 catalog_manager.cc:5484] T 5893a54b753f4d52a900d34140f6363d P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:06.290005 2213 leader_election.cc:288] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:06.290452 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9342fb2bff81431db52967c7bf4d331a" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:06.290673 29245 raft_consensus.cc:3037] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.291152 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9342fb2bff81431db52967c7bf4d331a" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:06.291309 29637 raft_consensus.cc:3037] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.293654 29245 raft_consensus.cc:2454] T 9342fb2bff81431db52967c7bf4d331a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:06.294111 29637 raft_consensus.cc:2454] T 9342fb2bff81431db52967c7bf4d331a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:06.295907 31963 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:06.296591 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 5893a54b753f4d52a900d34140f6363d with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:06.297641 29245 raft_consensus.cc:2932] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:06.295045 32030 leader_election.cc:302] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:06.300453 2213 raft_consensus.cc:2781] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:06.300997 2213 raft_consensus.cc:686] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:06.301630 2762 raft_consensus.cc:482] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:06.301828 2762 raft_consensus.cc:504] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.302657 2762 leader_election.cc:288] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:06.303200 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06abcab91b2430989e8d37175bc239d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:06.303277 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06abcab91b2430989e8d37175bc239d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:06.303458 29637 raft_consensus.cc:2454] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:06.303447 29506 raft_consensus.cc:2454] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:06.301297 2213 consensus_queue.cc:237] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:06.303836 29180 leader_election.cc:302] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:06.304100 2289 raft_consensus.cc:2781] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:06.304195 2289 raft_consensus.cc:482] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:06.304304 2289 raft_consensus.cc:3037] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.307612 2289 raft_consensus.cc:504] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.308439 2289 leader_election.cc:288] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:06.309002 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06abcab91b2430989e8d37175bc239d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:06.309235 29506 raft_consensus.cc:3037] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.311585 3023 tablet_replica.cc:330] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:06.311781 3023 raft_consensus.cc:2229] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:06.312220 3023 raft_consensus.cc:2258] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:06.312357 29506 raft_consensus.cc:2454] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:06.312899 29180 leader_election.cc:302] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:06.313153 3023 ts_tablet_manager.cc:1875] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:06.313277 2289 raft_consensus.cc:2781] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:06.313519 2289 raft_consensus.cc:686] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:06.313455 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06abcab91b2430989e8d37175bc239d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:06.313674 29637 raft_consensus.cc:3037] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.313947 2289 consensus_queue.cc:237] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:06.317570 29637 raft_consensus.cc:2454] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:06.318871 29103 catalog_manager.cc:5484] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:06.320904 3023 ts_tablet_manager.cc:1888] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:06.321197 3023 log.cc:1196] T 5893a54b753f4d52a900d34140f6363d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/5893a54b753f4d52a900d34140f6363d
I20240501 14:01:06.322052 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 5893a54b753f4d52a900d34140f6363d (table rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]) successfully deleted
I20240501 14:01:06.325016 2840 raft_consensus.cc:482] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:06.325191 2840 raft_consensus.cc:504] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.325982 2840 leader_election.cc:288] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:06.326853 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7c0ef199394de490b790a26532df3b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:06.327032 29636 raft_consensus.cc:2454] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:06.327576 32294 leader_election.cc:302] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:06.327827 2850 raft_consensus.cc:2781] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:06.327940 2850 raft_consensus.cc:482] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:06.328034 2850 raft_consensus.cc:3037] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.330224 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7c0ef199394de490b790a26532df3b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:06.330683 2850 raft_consensus.cc:504] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.331516 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7c0ef199394de490b790a26532df3b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:06.331748 29637 raft_consensus.cc:3037] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.331779 31963 raft_consensus.cc:2454] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:06.335111 29637 raft_consensus.cc:2454] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:06.335121 29103 catalog_manager.cc:5484] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:06.335568 32294 leader_election.cc:302] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:06.336118 2840 raft_consensus.cc:2781] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:06.336756 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7c0ef199394de490b790a26532df3b" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:06.337033 31963 raft_consensus.cc:3037] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:06.339581 31963 raft_consensus.cc:2454] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:06.340497 2850 leader_election.cc:288] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:06.345264 2840 raft_consensus.cc:686] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:06.345700 2840 consensus_queue.cc:237] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:06.349314 29096 catalog_manager.cc:5484] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:06.444173 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:06.455888 29506 raft_consensus.cc:1261] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.456112 29636 raft_consensus.cc:1261] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.456794 2289 consensus_queue.cc:1035] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.457741 3025 consensus_queue.cc:1035] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.489141 29103 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]). Expected version 1 got 0
I20240501 14:01:06.500656 3054 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:06.501297 3054 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:06.501909 29103 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]). Expected version 1 got 0
I20240501 14:01:06.519140 29103 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]). Expected version 1 got 0
I20240501 14:01:06.522825 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b] alter complete (version 1)
I20240501 14:01:06.511711 3053 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:06.526364 3053 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:06.538509 3055 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:06.539258 3055 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:06.539875 29103 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b]). Expected version 1 got 0
I20240501 14:01:06.632241 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:06.662039 3053 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:06.662635 3053 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:06.663414 3054 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:06.663834 3054 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:06.690693 3055 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:06.691182 3055 tablet.cc:1620] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:06.694648 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=15ffe5d78fdf4d4fb5d6c88818ff471b] alter complete (version 2)
I20240501 14:01:06.699676 2856 consensus_queue.cc:1035] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.721675 2632 consensus_queue.cc:1035] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.733681 1937 consensus_queue.cc:1035] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.737932 3028 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.782152 2840 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.785252 2632 consensus_queue.cc:1035] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:06.801661 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:06.821296 3053 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:06.822463 3054 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:06.822641 3055 tablet.cc:1722] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:06.833551 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=15ffe5d78fdf4d4fb5d6c88818ff471b] alter complete (version 3)
I20240501 14:01:06.881300 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:06.905382 32206 tablet_service.cc:1467] Processing CreateTablet for tablet c23d0b871aff4ce3850e45ae31097c03 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.905848 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23d0b871aff4ce3850e45ae31097c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.911655 32342 tablet_service.cc:1467] Processing CreateTablet for tablet c23d0b871aff4ce3850e45ae31097c03 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.912102 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23d0b871aff4ce3850e45ae31097c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.912351 29357 tablet_service.cc:1467] Processing CreateTablet for tablet c23d0b871aff4ce3850e45ae31097c03 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.912782 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23d0b871aff4ce3850e45ae31097c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.915128 29356 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b00f102874e17830f2e340255a11f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.915464 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b00f102874e17830f2e340255a11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.918859 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d23c77f4e432980759f1b4c2bebfa (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.919265 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d23c77f4e432980759f1b4c2bebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.921279 32207 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b00f102874e17830f2e340255a11f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.921675 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b00f102874e17830f2e340255a11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.925395 29487 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b00f102874e17830f2e340255a11f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.925822 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b00f102874e17830f2e340255a11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.929497 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d23c77f4e432980759f1b4c2bebfa (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.929916 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d23c77f4e432980759f1b4c2bebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.931486 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d23c77f4e432980759f1b4c2bebfa (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:06.931910 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d23c77f4e432980759f1b4c2bebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.939464 3088 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:06.944214 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:06.944484 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:06.948753 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet a06abcab91b2430989e8d37175bc239d
I20240501 14:01:06.949808 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet a06abcab91b2430989e8d37175bc239d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:06 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:06.950243 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet a06abcab91b2430989e8d37175bc239d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:06 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:06.950842 3088 tablet_bootstrap.cc:654] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.967626 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet a06abcab91b2430989e8d37175bc239d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:06 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:06.985354 3094 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:06.985915 3095 tablet_replica.cc:330] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:06.986390 3096 tablet_replica.cc:330] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:06.986966 3100 tablet_replica.cc:330] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:06.987422 3100 raft_consensus.cc:2229] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:06.987733 3100 raft_consensus.cc:2258] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:06.988867 3100 ts_tablet_manager.cc:1875] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:06.990586 3098 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:06.991261 3099 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:06.993623 3098 tablet_bootstrap.cc:654] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:06.997138 3096 raft_consensus.cc:2229] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:06.997687 3096 raft_consensus.cc:2258] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:06.998497 3095 raft_consensus.cc:2229] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:06.998755 3095 raft_consensus.cc:2258] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:06.990092 3101 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:07.003068 3101 tablet_bootstrap.cc:654] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.003242 3094 tablet_bootstrap.cc:654] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.004956 3099 tablet_bootstrap.cc:654] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.005724 3095 ts_tablet_manager.cc:1875] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:07.010409 3088 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:07.011040 3101 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:07.011435 3101 ts_tablet_manager.cc:1367] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 14:01:07.010730 3088 ts_tablet_manager.cc:1367] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240501 14:01:07.014305 3088 raft_consensus.cc:348] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.014456 3101 raft_consensus.cc:348] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.014703 3088 raft_consensus.cc:374] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.014816 3088 raft_consensus.cc:729] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.015599 3088 consensus_queue.cc:260] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.016391 3088 ts_tablet_manager.cc:1398] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:07.030772 2340 raft_consensus.cc:482] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:07.030947 2340 raft_consensus.cc:504] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.031836 2340 leader_election.cc:288] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:07.032779 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b00f102874e17830f2e340255a11f" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
W20240501 14:01:07.033758 29311 leader_election.cc:341] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:07.034468 3096 ts_tablet_manager.cc:1875] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:07.041589 3093 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:07.045210 3101 raft_consensus.cc:374] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.045300 3101 raft_consensus.cc:729] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.045485 3101 consensus_queue.cc:260] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.045979 3101 ts_tablet_manager.cc:1398] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.034s user 0.001s sys 0.000s
I20240501 14:01:07.029188 3095 ts_tablet_manager.cc:1888] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.033514 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b00f102874e17830f2e340255a11f" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:07.046360 3095 log.cc:1196] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/a06abcab91b2430989e8d37175bc239d
I20240501 14:01:07.046373 29506 raft_consensus.cc:2454] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 0.
I20240501 14:01:07.046939 3095 ts_tablet_manager.cc:1909] T a06abcab91b2430989e8d37175bc239d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:01:07.047760 29311 leader_election.cc:302] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters: 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:07.047960 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table_new_name [id=15ffe5d78fdf4d4fb5d6c88818ff471b]) successfully deleted
I20240501 14:01:07.048825 3098 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:07.049041 2340 raft_consensus.cc:2781] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:07.049129 2340 raft_consensus.cc:482] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:07.049206 2340 raft_consensus.cc:3037] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.051106 3098 ts_tablet_manager.cc:1367] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.003s
I20240501 14:01:07.052428 2340 raft_consensus.cc:504] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.055469 3098 raft_consensus.cc:348] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.055747 3098 raft_consensus.cc:374] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.056003 3098 raft_consensus.cc:729] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.062566 3088 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:07.062868 3098 consensus_queue.cc:260] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.064978 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:07.070621 3098 ts_tablet_manager.cc:1398] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240501 14:01:07.070956 3098 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:07.072002 3099 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:07.072218 3099 ts_tablet_manager.cc:1367] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.002s
I20240501 14:01:07.073029 3099 raft_consensus.cc:348] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.073297 3099 raft_consensus.cc:374] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.073412 3099 raft_consensus.cc:729] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.073714 3099 consensus_queue.cc:260] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.085214 3099 ts_tablet_manager.cc:1398] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240501 14:01:07.086334 32077 tablet_service.cc:1467] Processing CreateTablet for tablet df018395ca54471fb7b5eb46bdc84855 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]), partition=
I20240501 14:01:07.086788 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df018395ca54471fb7b5eb46bdc84855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.089243 29357 tablet_service.cc:1467] Processing CreateTablet for tablet df018395ca54471fb7b5eb46bdc84855 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]), partition=
I20240501 14:01:07.089568 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df018395ca54471fb7b5eb46bdc84855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.092635 29224 tablet_service.cc:1467] Processing CreateTablet for tablet df018395ca54471fb7b5eb46bdc84855 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]), partition=
I20240501 14:01:07.092949 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df018395ca54471fb7b5eb46bdc84855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.096732 2340 leader_election.cc:288] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:07.096823 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b00f102874e17830f2e340255a11f" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:07.097046 29506 raft_consensus.cc:3037] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.099642 29506 raft_consensus.cc:2454] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:07.100955 29311 leader_election.cc:302] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:07.104377 2340 raft_consensus.cc:2781] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:07.104540 2340 raft_consensus.cc:686] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:01:07.104867 2340 consensus_queue.cc:237] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:07.112864 3094 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:07.113134 3094 ts_tablet_manager.cc:1367] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.128s user 0.000s sys 0.004s
W20240501 14:01:07.117686 29310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 45215.8us
I20240501 14:01:07.121191 29103 catalog_manager.cc:5484] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:07.095826 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b00f102874e17830f2e340255a11f" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:07.130398 32227 raft_consensus.cc:3037] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.130965 3113 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:07.137461 32227 raft_consensus.cc:2454] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:01:07.138041 3112 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:07.139469 3112 tablet_bootstrap.cc:654] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.139505 3098 tablet_bootstrap.cc:654] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.139741 3088 tablet_bootstrap.cc:654] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.143402 3113 tablet_bootstrap.cc:654] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.155896 3112 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:07.157936 3112 ts_tablet_manager.cc:1367] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240501 14:01:07.158723 3112 raft_consensus.cc:348] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.159231 3112 raft_consensus.cc:374] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.159376 3112 raft_consensus.cc:729] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.159664 3112 consensus_queue.cc:260] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.162042 3112 ts_tablet_manager.cc:1398] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:01:07.168646 3098 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:07.168907 3098 ts_tablet_manager.cc:1367] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.004s
I20240501 14:01:07.169732 3098 raft_consensus.cc:348] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.174422 3098 raft_consensus.cc:374] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.174595 3098 raft_consensus.cc:729] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.174953 3098 consensus_queue.cc:260] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.177237 3094 raft_consensus.cc:348] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.178561 3098 ts_tablet_manager.cc:1398] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:01:07.178879 3098 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:07.178856 3094 raft_consensus.cc:374] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.178965 3094 raft_consensus.cc:729] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.179054 3093 tablet_bootstrap.cc:654] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.191084 3098 tablet_bootstrap.cc:654] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.193524 3113 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:07.193679 3113 ts_tablet_manager.cc:1367] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.000s
I20240501 14:01:07.194447 3113 raft_consensus.cc:348] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.194644 3113 raft_consensus.cc:374] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.194722 3113 raft_consensus.cc:729] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.194931 3113 consensus_queue.cc:260] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.195452 3113 ts_tablet_manager.cc:1398] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:07.196957 3098 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:07.197156 3098 ts_tablet_manager.cc:1367] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240501 14:01:07.197651 3088 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:07.197846 3088 ts_tablet_manager.cc:1367] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.000s
I20240501 14:01:07.197914 3098 raft_consensus.cc:348] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.198275 3098 raft_consensus.cc:374] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.198415 3098 raft_consensus.cc:729] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.198671 3098 consensus_queue.cc:260] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.199254 3098 ts_tablet_manager.cc:1398] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:07.188135 3094 consensus_queue.cc:260] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.203395 3094 ts_tablet_manager.cc:1398] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.090s user 0.003s sys 0.000s
I20240501 14:01:07.209348 3088 raft_consensus.cc:348] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.209596 3088 raft_consensus.cc:374] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.209694 3088 raft_consensus.cc:729] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.209899 3088 consensus_queue.cc:260] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.211124 3088 ts_tablet_manager.cc:1398] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:01:07.221577 3088 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:07.224239 3088 tablet_bootstrap.cc:654] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:07.236726 2632 raft_consensus.cc:482] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:07.236950 2632 raft_consensus.cc:504] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.237982 2632 leader_election.cc:288] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:07.238397 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df018395ca54471fb7b5eb46bdc84855" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:07.239147 3093 tablet_bootstrap.cc:492] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:07.239097 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df018395ca54471fb7b5eb46bdc84855" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
W20240501 14:01:07.239185 32032 leader_election.cc:341] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:07.239316 29245 raft_consensus.cc:2454] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:07.239653 3093 ts_tablet_manager.cc:1367] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.199s user 0.005s sys 0.000s
I20240501 14:01:07.239676 32030 leader_election.cc:302] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:01:07.240667 2632 raft_consensus.cc:2781] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:07.240799 2632 raft_consensus.cc:482] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:07.241091 2632 raft_consensus.cc:3037] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.240751 3093 raft_consensus.cc:348] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.241336 3093 raft_consensus.cc:374] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.241503 3093 raft_consensus.cc:729] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.241885 3093 consensus_queue.cc:260] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.243971 2632 raft_consensus.cc:504] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.244844 2632 leader_election.cc:288] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:07.245158 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df018395ca54471fb7b5eb46bdc84855" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
W20240501 14:01:07.245683 32032 leader_election.cc:341] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:07.246311 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df018395ca54471fb7b5eb46bdc84855" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:07.246477 29245 raft_consensus.cc:3037] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.248200 29245 raft_consensus.cc:2454] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:07.249208 32030 leader_election.cc:302] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:01:07.249668 2632 raft_consensus.cc:2781] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:07.250821 3093 ts_tablet_manager.cc:1398] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:01:07.251385 2632 raft_consensus.cc:686] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:07.251677 2632 consensus_queue.cc:237] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.256615 29097 catalog_manager.cc:5484] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:07.260202 3096 ts_tablet_manager.cc:1888] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.262987 3096 log.cc:1196] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/a06abcab91b2430989e8d37175bc239d
I20240501 14:01:07.261735 3100 ts_tablet_manager.cc:1888] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.263397 3088 tablet_bootstrap.cc:492] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:07.263610 3088 ts_tablet_manager.cc:1367] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.003s
I20240501 14:01:07.264075 3100 log.cc:1196] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/a06abcab91b2430989e8d37175bc239d
I20240501 14:01:07.264293 3088 raft_consensus.cc:348] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.264495 3088 raft_consensus.cc:374] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:07.264557 3088 raft_consensus.cc:729] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:07.264756 3088 consensus_queue.cc:260] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:07.265275 3088 ts_tablet_manager.cc:1398] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:07.269074 3096 ts_tablet_manager.cc:1909] T a06abcab91b2430989e8d37175bc239d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:01:07.269965 3100 ts_tablet_manager.cc:1909] T a06abcab91b2430989e8d37175bc239d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:07.274928 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table_new_name [id=15ffe5d78fdf4d4fb5d6c88818ff471b]) successfully deleted
I20240501 14:01:07.277268 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet a06abcab91b2430989e8d37175bc239d (table rebalancer_dynamic_table_new_name [id=15ffe5d78fdf4d4fb5d6c88818ff471b]) successfully deleted
I20240501 14:01:07.320290 2808 raft_consensus.cc:482] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:07.320451 2808 raft_consensus.cc:504] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.321296 2808 leader_election.cc:288] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:07.321672 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d23c77f4e432980759f1b4c2bebfa" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:07.321884 29636 raft_consensus.cc:2454] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:07.322086 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d23c77f4e432980759f1b4c2bebfa" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:07.322314 31963 raft_consensus.cc:2454] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:07.322712 32161 leader_election.cc:302] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:07.322962 2808 raft_consensus.cc:2781] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:07.323089 2808 raft_consensus.cc:482] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:07.323172 2808 raft_consensus.cc:3037] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.326668 2808 raft_consensus.cc:504] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.327571 2808 leader_election.cc:288] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:07.329320 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d23c77f4e432980759f1b4c2bebfa" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:07.329504 31963 raft_consensus.cc:3037] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.332190 31963 raft_consensus.cc:2454] T 6a9d23c77f4e432980759f1b4c2bebfa P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:07.333026 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d23c77f4e432980759f1b4c2bebfa" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:07.333250 29636 raft_consensus.cc:3037] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.335430 29636 raft_consensus.cc:2454] T 6a9d23c77f4e432980759f1b4c2bebfa P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:07.337204 32161 leader_election.cc:302] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:07.339382 2708 raft_consensus.cc:2781] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:07.360929 2708 raft_consensus.cc:686] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:07.361378 32077 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:01:07.361442 2708 consensus_queue.cc:237] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.363512 3138 raft_consensus.cc:482] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:07.363713 3138 raft_consensus.cc:504] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.365374 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23d0b871aff4ce3850e45ae31097c03" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:07.365634 32362 raft_consensus.cc:2454] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:07.366125 32162 leader_election.cc:302] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:07.366714 3138 leader_election.cc:288] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:07.366935 3138 raft_consensus.cc:2781] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:07.367173 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23d0b871aff4ce3850e45ae31097c03" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:07.367389 29374 raft_consensus.cc:2454] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:07.369084 3138 raft_consensus.cc:482] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:07.369244 3138 raft_consensus.cc:3037] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.369470 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:07.384177 3138 raft_consensus.cc:504] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.385735 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23d0b871aff4ce3850e45ae31097c03" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:07.385972 32362 raft_consensus.cc:3037] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.394323 32362 raft_consensus.cc:2454] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:07.394943 32162 leader_election.cc:302] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:07.395273 2808 raft_consensus.cc:2781] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:07.395529 2808 raft_consensus.cc:686] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:07.395902 2808 consensus_queue.cc:237] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:07.396150 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23d0b871aff4ce3850e45ae31097c03" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:07.396317 29374 raft_consensus.cc:3037] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.399701 29374 raft_consensus.cc:2454] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:07.400355 3138 leader_election.cc:288] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:07.407171 29374 raft_consensus.cc:3037] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:07.407160 29245 raft_consensus.cc:1261] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.409329 2213 consensus_queue.cc:1035] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.414857 29374 raft_consensus.cc:1261] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.416004 2213 consensus_queue.cc:1035] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.426841 29097 catalog_manager.cc:5484] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:07.443056 29097 catalog_manager.cc:5484] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:07.453016 3148 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:07.453822 3148 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:07.455073 3149 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:07.455751 3149 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:07.456066 29097 catalog_manager.cc:5576] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201) does not have the latest schema for tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]). Expected version 1 got 0
I20240501 14:01:07.458846 29224 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:07.464251 29487 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:07.471848 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678] alter complete (version 1)
I20240501 14:01:07.482053 29097 catalog_manager.cc:5576] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595) does not have the latest schema for tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]). Expected version 1 got 0
I20240501 14:01:07.498217 29097 catalog_manager.cc:5576] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595) does not have the latest schema for tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678]). Expected version 1 got 0
I20240501 14:01:07.494963 3151 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:07.506532 3151 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:01:07.578357 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:07.598950 2340 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.600350 3148 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:07.600864 3148 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:07.608515 3149 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:07.609196 3149 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:07.624545 32342 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:07.628932 29617 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:07.628929 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=2f37b2f8198a40e4985b3423df576678] alter complete (version 2)
I20240501 14:01:07.649596 3151 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:07.650161 3151 tablet.cc:1620] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: MemRowSet was empty: no flush needed.
I20240501 14:01:07.681201 2920 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.687827 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:07.692111 31943 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:07.711055 29357 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:07.732529 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:07.743110 3148 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:07.744659 3149 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:07.759210 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=2f37b2f8198a40e4985b3423df576678] alter complete (version 3)
I20240501 14:01:07.762427 3151 tablet.cc:1722] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:07.833683 2808 consensus_queue.cc:1035] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.838238 3138 consensus_queue.cc:1035] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.859966 29505 consensus_queue.cc:237] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:07.861778 29097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:07.861984 29097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:07.866154 29097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet df018395ca54471fb7b5eb46bdc84855
I20240501 14:01:07.867085 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet df018395ca54471fb7b5eb46bdc84855 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:07 UTC) from {username='slave'} at 127.0.0.1:51134
I20240501 14:01:07.869411 32077 tablet_service.cc:1514] Processing DeleteTablet for tablet df018395ca54471fb7b5eb46bdc84855 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:07 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:07.870890 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet df018395ca54471fb7b5eb46bdc84855 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:07 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:07.871681 3186 tablet_replica.cc:330] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:01:07.871886 3186 raft_consensus.cc:2229] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:07.872143 3186 raft_consensus.cc:2258] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:07.874897 3186 ts_tablet_manager.cc:1875] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:07.879828 29374 raft_consensus.cc:1261] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:07.879906 32227 raft_consensus.cc:1261] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:07.880416 2616 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:07.881651 2964 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:07.881814 3186 ts_tablet_manager.cc:1888] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.881907 3186 log.cc:1196] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/df018395ca54471fb7b5eb46bdc84855
I20240501 14:01:07.882253 3186 ts_tablet_manager.cc:1909] T df018395ca54471fb7b5eb46bdc84855 P b87d188552494b4590afc37a85b98c35: Deleting consensus metadata
I20240501 14:01:07.883303 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table_new_name [id=2f37b2f8198a40e4985b3423df576678]) successfully deleted
I20240501 14:01:07.892133 2616 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:07.901822 3188 tablet_replica.cc:330] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:07.903257 32227 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:07.913794 3188 raft_consensus.cc:2229] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:07.913997 3188 raft_consensus.cc:2258] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:07.914314 3187 tablet_replica.cc:330] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:07.914599 3187 raft_consensus.cc:2229] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:07.915027 3187 raft_consensus.cc:2258] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:07.915915 3188 ts_tablet_manager.cc:1875] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:07.917474 3187 ts_tablet_manager.cc:1875] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:07.917811 29103 catalog_manager.cc:5484] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:07.922540 29376 raft_consensus.cc:2932] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:07.929800 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8d8249c87f42faa9a7464369048090 with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:07.931720 3188 ts_tablet_manager.cc:1888] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.931830 3188 log.cc:1196] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/df018395ca54471fb7b5eb46bdc84855
I20240501 14:01:07.932210 3188 ts_tablet_manager.cc:1909] T df018395ca54471fb7b5eb46bdc84855 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:01:07.933171 3188 tablet_replica.cc:330] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:07.933362 3188 raft_consensus.cc:2229] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:07.933563 3188 raft_consensus.cc:2258] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:07.933609 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table_new_name [id=2f37b2f8198a40e4985b3423df576678]) successfully deleted
I20240501 14:01:07.934185 3188 ts_tablet_manager.cc:1875] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:07.939281 3187 ts_tablet_manager.cc:1888] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:07.939435 3187 log.cc:1196] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/df018395ca54471fb7b5eb46bdc84855
I20240501 14:01:07.940133 3187 ts_tablet_manager.cc:1909] T df018395ca54471fb7b5eb46bdc84855 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:07.941448 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet df018395ca54471fb7b5eb46bdc84855 (table rebalancer_dynamic_table_new_name [id=2f37b2f8198a40e4985b3423df576678]) successfully deleted
I20240501 14:01:07.945135 3188 ts_tablet_manager.cc:1888] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:07.945333 3188 log.cc:1196] T 5a8d8249c87f42faa9a7464369048090 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/5a8d8249c87f42faa9a7464369048090
I20240501 14:01:07.947917 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 5a8d8249c87f42faa9a7464369048090 (table rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]) successfully deleted
W20240501 14:01:07.952422 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 49895.7us
I20240501 14:01:07.961416 3139 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.964162 3138 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:07.969144 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:07.982447 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 0c8060687b634344a68df56fc61acbfe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]), partition=
I20240501 14:01:07.982864 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c8060687b634344a68df56fc61acbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.982894 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 0c8060687b634344a68df56fc61acbfe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]), partition=
I20240501 14:01:07.983173 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c8060687b634344a68df56fc61acbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.985275 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 0c8060687b634344a68df56fc61acbfe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]), partition=
I20240501 14:01:07.985596 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c8060687b634344a68df56fc61acbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.002032 3206 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:08.004667 3206 tablet_bootstrap.cc:654] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.014676 3205 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:08.014708 3206 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:08.015328 3206 ts_tablet_manager.cc:1367] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 14:01:08.016481 3206 raft_consensus.cc:348] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.017930 3206 raft_consensus.cc:374] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.018162 3206 raft_consensus.cc:729] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.018702 3206 consensus_queue.cc:260] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.021736 3206 ts_tablet_manager.cc:1398] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:08.028805 3205 tablet_bootstrap.cc:654] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.038996 3205 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:08.039193 3205 ts_tablet_manager.cc:1367] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.005s
I20240501 14:01:08.041854 3205 raft_consensus.cc:348] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.042443 3205 raft_consensus.cc:374] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.042896 3205 raft_consensus.cc:729] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.043560 3205 consensus_queue.cc:260] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.046022 3205 ts_tablet_manager.cc:1398] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:01:08.060240 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:08.077514 3204 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:08.080785 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 10e1775ccab942ff857d22e89d99764d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.081502 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e1775ccab942ff857d22e89d99764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.081851 31943 tablet_service.cc:1467] Processing CreateTablet for tablet d2cf9b7f68034fcb80cf620687ca4a81 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.082340 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cf9b7f68034fcb80cf620687ca4a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.083490 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 10e1775ccab942ff857d22e89d99764d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.083914 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e1775ccab942ff857d22e89d99764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.084434 3204 tablet_bootstrap.cc:654] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.087280 32207 tablet_service.cc:1467] Processing CreateTablet for tablet d2cf9b7f68034fcb80cf620687ca4a81 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.087697 32077 tablet_service.cc:1467] Processing CreateTablet for tablet b516ab873c134b79a4424f5ffbc471fb (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.087849 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cf9b7f68034fcb80cf620687ca4a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.088186 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b516ab873c134b79a4424f5ffbc471fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.091374 31944 tablet_service.cc:1467] Processing CreateTablet for tablet b516ab873c134b79a4424f5ffbc471fb (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.091854 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b516ab873c134b79a4424f5ffbc471fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.093475 32340 tablet_service.cc:1467] Processing CreateTablet for tablet d2cf9b7f68034fcb80cf620687ca4a81 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.093959 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2cf9b7f68034fcb80cf620687ca4a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.098456 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 10e1775ccab942ff857d22e89d99764d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.098899 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e1775ccab942ff857d22e89d99764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.101112 29357 tablet_service.cc:1467] Processing CreateTablet for tablet b516ab873c134b79a4424f5ffbc471fb (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.101492 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b516ab873c134b79a4424f5ffbc471fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.104362 3212 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:08.107486 3212 tablet_bootstrap.cc:654] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.111670 3205 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:08.114240 3205 tablet_bootstrap.cc:654] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.128798 3204 tablet_bootstrap.cc:492] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:08.128968 3204 ts_tablet_manager.cc:1367] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240501 14:01:08.130345 3204 raft_consensus.cc:348] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.130594 3204 raft_consensus.cc:374] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.130653 3204 raft_consensus.cc:729] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.130877 3204 consensus_queue.cc:260] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.131732 3204 ts_tablet_manager.cc:1398] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:08.132031 3204 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:08.134397 3204 tablet_bootstrap.cc:654] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.138885 3205 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:08.139614 3205 ts_tablet_manager.cc:1367] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.001s
I20240501 14:01:08.143296 3205 raft_consensus.cc:348] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.147637 3205 raft_consensus.cc:374] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.148084 3205 raft_consensus.cc:729] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.148415 3205 consensus_queue.cc:260] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.148941 3216 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:08.152633 3205 ts_tablet_manager.cc:1398] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:01:08.153102 3216 tablet_bootstrap.cc:654] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.161998 3219 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:08.164597 3219 tablet_bootstrap.cc:654] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.169971 3204 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:08.170193 3204 ts_tablet_manager.cc:1367] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240501 14:01:08.171098 3204 raft_consensus.cc:348] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.171353 3204 raft_consensus.cc:374] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.171466 3204 raft_consensus.cc:729] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.171758 3204 consensus_queue.cc:260] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.172464 3204 ts_tablet_manager.cc:1398] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:08.175237 3212 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:08.175478 3212 ts_tablet_manager.cc:1367] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20240501 14:01:08.176590 3212 raft_consensus.cc:348] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.176885 3212 raft_consensus.cc:374] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.177068 3212 raft_consensus.cc:729] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.177359 3212 consensus_queue.cc:260] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.179318 3212 ts_tablet_manager.cc:1398] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:08.181309 3212 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:08.181414 3219 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:08.182534 3219 ts_tablet_manager.cc:1367] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240501 14:01:08.184517 3212 tablet_bootstrap.cc:654] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.187291 3222 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:08.186959 3219 raft_consensus.cc:348] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.187944 3219 raft_consensus.cc:374] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.188097 3219 raft_consensus.cc:729] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.190851 3219 consensus_queue.cc:260] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.193738 3079 raft_consensus.cc:482] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.193964 3079 raft_consensus.cc:504] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.194590 3219 ts_tablet_manager.cc:1398] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:01:08.194872 3079 leader_election.cc:288] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.195223 3138 raft_consensus.cc:482] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.195399 3138 raft_consensus.cc:504] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.196202 3138 leader_election.cc:288] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.197201 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:08.197511 31965 raft_consensus.cc:2454] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
I20240501 14:01:08.198450 3222 tablet_bootstrap.cc:654] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.198848 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:01:08.199563 32162 leader_election.cc:341] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.200029 32163 leader_election.cc:302] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:08.200594 3138 raft_consensus.cc:2781] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:08.200740 3138 raft_consensus.cc:482] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:08.200878 3138 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.203903 3222 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:08.204023 3222 ts_tablet_manager.cc:1367] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:01:08.204674 3222 raft_consensus.cc:348] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.205046 3222 raft_consensus.cc:374] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.205269 3222 raft_consensus.cc:729] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.205947 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:01:08.207463 32032 leader_election.cc:341] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.209357 3222 consensus_queue.cc:260] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.211450 3222 ts_tablet_manager.cc:1398] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:08.212616 3212 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:08.212896 3212 ts_tablet_manager.cc:1367] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240501 14:01:08.214025 3212 raft_consensus.cc:348] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.214351 3212 raft_consensus.cc:374] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.214490 3212 raft_consensus.cc:729] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.214833 3212 consensus_queue.cc:260] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.217905 2762 raft_consensus.cc:482] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.218046 2762 raft_consensus.cc:504] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.218607 2762 leader_election.cc:288] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:08.219591 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c8060687b634344a68df56fc61acbfe" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:08.219874 32096 raft_consensus.cc:2454] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:08.219763 3138 raft_consensus.cc:504] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.220436 29181 leader_election.cc:302] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:08.220772 2762 raft_consensus.cc:2781] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:08.220845 2762 raft_consensus.cc:482] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:08.220911 2762 raft_consensus.cc:3037] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.223014 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:08.223858 3217 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
W20240501 14:01:08.225083 32162 leader_election.cc:341] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.226110 2762 raft_consensus.cc:504] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.227141 2762 leader_election.cc:288] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
W20240501 14:01:08.228713 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 58157us
I20240501 14:01:08.229235 3212 ts_tablet_manager.cc:1398] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:01:08.229790 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c8060687b634344a68df56fc61acbfe" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:08.230033 29507 raft_consensus.cc:3037] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.232765 29507 raft_consensus.cc:2454] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:08.233047 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c8060687b634344a68df56fc61acbfe" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:08.233355 32096 raft_consensus.cc:3037] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.233640 3138 leader_election.cc:288] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.229790 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c8060687b634344a68df56fc61acbfe" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:08.234077 29505 raft_consensus.cc:2362] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1. Re-sending same reply.
I20240501 14:01:08.236313 3217 tablet_bootstrap.cc:654] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.236711 32096 raft_consensus.cc:2454] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:08.237617 29180 leader_election.cc:302] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
I20240501 14:01:08.237920 2289 raft_consensus.cc:2781] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:08.238049 2289 raft_consensus.cc:686] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:08.238298 2289 consensus_queue.cc:237] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:08.246316 3216 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:08.246522 3216 ts_tablet_manager.cc:1367] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.010s
I20240501 14:01:08.247329 3216 raft_consensus.cc:348] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.247529 3216 raft_consensus.cc:374] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.247681 3216 raft_consensus.cc:729] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.247915 3216 consensus_queue.cc:260] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.248618 3216 ts_tablet_manager.cc:1398] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:08.248950 3216 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:08.250540 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:08.250823 31965 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.258478 3216 tablet_bootstrap.cc:654] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:08.259406 3028 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.269217 3028 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.270462 31965 raft_consensus.cc:2454] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:08.270447 3028 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:08.271131 32163 leader_election.cc:302] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:08.271306 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:08.271544 29636 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:08.271690 3139 raft_consensus.cc:2781] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:08.271958 3139 raft_consensus.cc:686] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:08.272094 32294 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:08.272527 3028 raft_consensus.cc:2781] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:08.272464 3139 consensus_queue.cc:237] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.274364 3028 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:08.274292 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:08.274536 3028 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.274519 29246 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:08.277921 3028 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.279415 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:08.279592 29636 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.280620 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:08.280802 29246 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.282373 29636 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:08.282975 32294 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:08.283303 2840 raft_consensus.cc:2781] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:08.284109 2840 raft_consensus.cc:686] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:08.284461 2840 consensus_queue.cc:237] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:08.285688 3028 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:08.300715 29246 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:08.326505 3090 raft_consensus.cc:482] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.326807 3090 raft_consensus.cc:504] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.327896 3090 leader_election.cc:288] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:08.326526 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:08.329171 32097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:08.329556 32097 raft_consensus.cc:2454] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:08.330345 31900 leader_election.cc:302] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:08.330853 3090 raft_consensus.cc:2781] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:08.331070 3090 raft_consensus.cc:482] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:08.331248 3090 raft_consensus.cc:3037] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.343907 3090 raft_consensus.cc:504] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.344861 3090 leader_election.cc:288] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:08.345760 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:08.346109 32096 raft_consensus.cc:3037] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.351512 32096 raft_consensus.cc:2454] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:08.352398 31900 leader_election.cc:302] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:08.353673 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:08.362551 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:08.363113 3217 tablet_bootstrap.cc:492] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:08.363242 3217 ts_tablet_manager.cc:1367] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.140s user 0.003s sys 0.002s
I20240501 14:01:08.364038 3217 raft_consensus.cc:348] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.364689 3090 raft_consensus.cc:2781] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:08.364713 29103 catalog_manager.cc:5484] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:08.365357 3090 raft_consensus.cc:686] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:08.365785 3090 consensus_queue.cc:237] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.367494 3217 raft_consensus.cc:374] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.368865 29097 catalog_manager.cc:5484] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:08.369153 3217 raft_consensus.cc:729] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:08.369491 3217 consensus_queue.cc:260] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:01:08.372366 31900 leader_election.cc:341] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.374330 3217 ts_tablet_manager.cc:1398] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:01:08.379060 29103 catalog_manager.cc:5484] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
W20240501 14:01:08.390353 31900 leader_election.cc:341] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.397473 3216 tablet_bootstrap.cc:492] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:08.397821 3216 ts_tablet_manager.cc:1367] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.032s
I20240501 14:01:08.399112 3216 raft_consensus.cc:348] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.399300 29103 catalog_manager.cc:5484] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:08.399919 3216 raft_consensus.cc:374] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:08.400130 3216 raft_consensus.cc:729] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
W20240501 14:01:08.404271 32032 leader_election.cc:341] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:08.404466 32032 leader_election.cc:302] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35
I20240501 14:01:08.404861 2632 raft_consensus.cc:2726] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:08.406459 3216 consensus_queue.cc:260] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.407176 3216 ts_tablet_manager.cc:1398] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:01:08.628027 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:08.650799 32095 raft_consensus.cc:1261] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:08.656121 2289 consensus_queue.cc:1035] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:08.672686 29101 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]). Expected version 1 got 0
I20240501 14:01:08.690265 29507 raft_consensus.cc:1261] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:08.691061 2289 consensus_queue.cc:1035] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:08.693974 2840 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:08.729652 3280 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:08.730430 3280 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:08.736320 3281 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:08.738672 3281 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:08.739773 29104 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]). Expected version 1 got 0
I20240501 14:01:08.747591 29103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61] alter complete (version 1)
I20240501 14:01:08.764288 29103 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]). Expected version 1 got 0
I20240501 14:01:08.770836 3234 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:08.811305 32362 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.816869 3234 raft_consensus.cc:482] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.817116 3234 raft_consensus.cc:504] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.818115 3234 leader_election.cc:288] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.822019 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:08.822393 31965 raft_consensus.cc:2454] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:08.823668 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:08.823968 3138 consensus_queue.cc:1035] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:08.827111 3291 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:08.827702 3291 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:08.832481 32077 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:08.834666 32296 leader_election.cc:302] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:08.843478 29103 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61]). Expected version 1 got 0
I20240501 14:01:08.854995 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:08.861409 3236 raft_consensus.cc:482] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:08.861593 3236 raft_consensus.cc:504] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.862331 3236 leader_election.cc:288] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.863109 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:08.863305 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b516ab873c134b79a4424f5ffbc471fb" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:08.863603 32095 raft_consensus.cc:2379] T b516ab873c134b79a4424f5ffbc471fb P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b87d188552494b4590afc37a85b98c35 in current term 1: Already voted for candidate 9903ed35bca144bcb74e9bf5102117e7 in this term.
I20240501 14:01:08.863974 29312 leader_election.cc:302] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87d188552494b4590afc37a85b98c35; no voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:08.864421 3236 raft_consensus.cc:3037] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:08.867202 3236 raft_consensus.cc:2726] T b516ab873c134b79a4424f5ffbc471fb P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:08.873162 3280 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:08.873662 3280 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: MemRowSet was empty: no flush needed.
I20240501 14:01:08.874629 2840 raft_consensus.cc:2781] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:08.874905 2840 raft_consensus.cc:482] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6a446ac802a7457d9f27076781f21d79)
I20240501 14:01:08.875030 2840 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:08.877179 3281 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:08.877686 3281 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:08.878259 2840 raft_consensus.cc:504] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:08.879124 2840 leader_election.cc:288] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:08.879403 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:08.907730 29100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=53379e4c938649d38a2f40cb6aa2ab61] alter complete (version 2)
I20240501 14:01:08.921290 32361 raft_consensus.cc:1226] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Rejecting Update request from peer 6a446ac802a7457d9f27076781f21d79 for earlier term 1. Current term is 2. Ops: []
I20240501 14:01:08.947185 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:08.952531 29487 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:08.992678 32207 tablet_service.cc:1467] Processing CreateTablet for tablet c05d6d2f1f1140c7bea1e084574b51fe (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.993235 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05d6d2f1f1140c7bea1e084574b51fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.993438 32077 tablet_service.cc:1467] Processing CreateTablet for tablet c05d6d2f1f1140c7bea1e084574b51fe (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:08.993973 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05d6d2f1f1140c7bea1e084574b51fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.999893 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 798c9c25344d4beb9a079fb776044a17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.000183 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 798c9c25344d4beb9a079fb776044a17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.000355 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798c9c25344d4beb9a079fb776044a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.001121 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 798c9c25344d4beb9a079fb776044a17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.001549 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798c9c25344d4beb9a079fb776044a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.001853 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.002328 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.010838 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.011236 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.016500 29100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:09.018487 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.018931 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b4f0fc35a2401a9b80c3d590e6dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.019905 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798c9c25344d4beb9a079fb776044a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.021626 3316 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:09.023262 3316 tablet_bootstrap.cc:654] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.035518 3316 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:09.035805 3316 ts_tablet_manager.cc:1367] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 14:01:09.037040 3316 raft_consensus.cc:348] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.037416 3316 raft_consensus.cc:374] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.037531 3316 raft_consensus.cc:729] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.037819 3316 consensus_queue.cc:260] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.038010 3280 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:09.039440 3281 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:09.040445 3316 ts_tablet_manager.cc:1398] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:09.056299 3315 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:09.058250 3315 tablet_bootstrap.cc:654] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.060081 3319 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:09.062333 3319 tablet_bootstrap.cc:654] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.064066 3320 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:09.066874 3320 tablet_bootstrap.cc:654] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.072351 3321 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
W20240501 14:01:09.074872 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 131751us
I20240501 14:01:09.085783 3164 raft_consensus.cc:482] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.085959 3164 raft_consensus.cc:504] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.086959 3164 leader_election.cc:288] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:09.088086 29100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=53379e4c938649d38a2f40cb6aa2ab61] alter complete (version 3)
I20240501 14:01:09.088016 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798c9c25344d4beb9a079fb776044a17" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
W20240501 14:01:09.097944 29571 leader_election.cc:341] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:09.103236 3315 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:09.105278 3315 ts_tablet_manager.cc:1367] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20240501 14:01:09.106097 3315 raft_consensus.cc:348] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.110216 3315 raft_consensus.cc:374] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.111281 3321 tablet_bootstrap.cc:654] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.111526 32342 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:09.116022 3319 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:09.120072 3319 ts_tablet_manager.cc:1367] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20240501 14:01:09.121915 3315 raft_consensus.cc:729] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.121894 3319 raft_consensus.cc:348] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.122035 29617 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:09.129757 3323 consensus_queue.cc:1046] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.130154 3323 raft_consensus.cc:3032] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:09.130329 3323 raft_consensus.cc:729] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:09.130515 3319 raft_consensus.cc:374] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.130860 3319 raft_consensus.cc:729] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.130765 3323 consensus_queue.cc:260] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.131274 3323 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:09.131654 3319 consensus_queue.cc:260] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.136758 3319 ts_tablet_manager.cc:1398] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:01:09.153450 29357 tablet_service.cc:1467] Processing CreateTablet for tablet c05d6d2f1f1140c7bea1e084574b51fe (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:09.153930 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05d6d2f1f1140c7bea1e084574b51fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.134963 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798c9c25344d4beb9a079fb776044a17" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:09.160940 3315 consensus_queue.cc:260] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.162094 29244 raft_consensus.cc:2454] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:09.162339 3321 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:09.162453 3315 ts_tablet_manager.cc:1398] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.057s user 0.000s sys 0.002s
I20240501 14:01:09.162561 3321 ts_tablet_manager.cc:1367] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.014s
I20240501 14:01:09.163414 3321 raft_consensus.cc:348] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.163666 3321 raft_consensus.cc:374] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.163748 3321 raft_consensus.cc:729] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.164008 3321 consensus_queue.cc:260] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.164793 3321 ts_tablet_manager.cc:1398] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:09.168558 3321 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:09.169188 3315 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:09.170369 3321 tablet_bootstrap.cc:654] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.171738 3315 tablet_bootstrap.cc:654] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.190909 3315 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
W20240501 14:01:09.191634 31900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 316878us
I20240501 14:01:09.192957 29100 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table_new_name [id=53379e4c938649d38a2f40cb6aa2ab61]). Expected version 3 got 1
I20240501 14:01:09.191180 3315 ts_tablet_manager.cc:1367] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.002s
I20240501 14:01:09.199558 29100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:09.199714 29100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:09.198741 3315 raft_consensus.cc:348] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.207626 3315 raft_consensus.cc:374] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.207746 3315 raft_consensus.cc:729] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.208043 3315 consensus_queue.cc:260] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.208866 3315 ts_tablet_manager.cc:1398] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:01:09.213557 29100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 0c8060687b634344a68df56fc61acbfe
I20240501 14:01:09.214538 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c8060687b634344a68df56fc61acbfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:09 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:09.215447 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c8060687b634344a68df56fc61acbfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:09 UTC) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:09.215704 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c8060687b634344a68df56fc61acbfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:09 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:09.226049 3318 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:09.231884 3331 tablet_replica.cc:330] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:09.235457 3331 raft_consensus.cc:2229] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.237356 3331 raft_consensus.cc:2258] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:09.231115 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 121091us
I20240501 14:01:09.236479 3291 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:09.243855 3291 tablet.cc:1620] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:09.254230 3332 tablet_replica.cc:330] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:09.254603 3332 raft_consensus.cc:2229] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:09.255398 3332 raft_consensus.cc:2258] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.256518 3332 ts_tablet_manager.cc:1875] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:09.259670 29570 leader_election.cc:302] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:09.260128 3164 raft_consensus.cc:2781] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:09.260215 3164 raft_consensus.cc:482] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:09.260273 3164 raft_consensus.cc:3037] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.261601 3191 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.261799 3191 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.262660 3191 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:09.263304 3164 raft_consensus.cc:504] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.264683 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798c9c25344d4beb9a079fb776044a17" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:09.264899 32226 raft_consensus.cc:3037] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.265959 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:01:09.266582 29442 leader_election.cc:341] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:09.267740 32226 raft_consensus.cc:2454] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:09.267923 3291 tablet.cc:1722] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:09.268388 29571 leader_election.cc:302] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:09.268661 2856 raft_consensus.cc:2781] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:09.270162 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:09.270499 29244 raft_consensus.cc:2454] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:01:09.270946 29440 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters: 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:09.271278 3191 raft_consensus.cc:2781] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:09.271405 3191 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:09.271494 3191 raft_consensus.cc:3037] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.271803 3164 leader_election.cc:288] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:09.276095 3191 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.277001 3191 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:09.277431 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:09.277758 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:09.277954 29244 raft_consensus.cc:3037] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.281353 29244 raft_consensus.cc:2454] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:01:09.282238 29440 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba; no voters:
W20240501 14:01:09.282487 29442 leader_election.cc:341] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:09.282601 3193 raft_consensus.cc:2781] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:09.287562 3193 raft_consensus.cc:686] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:01:09.288283 3193 consensus_queue.cc:237] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.292258 2856 raft_consensus.cc:686] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:09.292639 2856 consensus_queue.cc:237] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.303336 29103 catalog_manager.cc:5484] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:09.306725 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798c9c25344d4beb9a079fb776044a17" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:09.306921 29244 raft_consensus.cc:3037] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.310253 29244 raft_consensus.cc:2454] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:09.314560 3332 ts_tablet_manager.cc:1888] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:09.314878 3332 log.cc:1196] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/0c8060687b634344a68df56fc61acbfe
I20240501 14:01:09.315343 3332 ts_tablet_manager.cc:1909] T 0c8060687b634344a68df56fc61acbfe P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting consensus metadata
I20240501 14:01:09.314126 3331 ts_tablet_manager.cc:1875] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:09.316558 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table_new_name [id=53379e4c938649d38a2f40cb6aa2ab61]) successfully deleted
I20240501 14:01:09.317957 3326 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:09.317211 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:09.322008 3326 tablet_bootstrap.cc:654] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.342783 3320 tablet_bootstrap.cc:492] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:09.343190 3320 ts_tablet_manager.cc:1367] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.279s user 0.012s sys 0.013s
I20240501 14:01:09.324030 29100 catalog_manager.cc:5484] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:09.358855 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]), partition=
I20240501 14:01:09.358892 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]), partition=
I20240501 14:01:09.359241 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ae82d0e6848b2baf967306b1a068d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.359323 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ae82d0e6848b2baf967306b1a068d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.360239 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]), partition=
I20240501 14:01:09.360587 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ae82d0e6848b2baf967306b1a068d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.366989 3331 ts_tablet_manager.cc:1888] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:09.367997 3316 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:09.369419 3316 tablet_bootstrap.cc:654] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.375133 3320 raft_consensus.cc:348] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.375512 3320 raft_consensus.cc:374] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.375698 3320 raft_consensus.cc:729] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.376685 3320 consensus_queue.cc:260] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.379398 29357 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:09.384505 3333 tablet_replica.cc:330] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:09.384824 3333 raft_consensus.cc:2229] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.388252 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2cf9b7f68034fcb80cf620687ca4a81" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:09.388478 31964 raft_consensus.cc:3037] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:09.390690 3333 raft_consensus.cc:2258] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.391281 3321 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:09.368429 3319 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:09.392588 31964 raft_consensus.cc:2454] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:01:09.392693 3316 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:09.392838 3316 ts_tablet_manager.cc:1367] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20240501 14:01:09.391487 3321 ts_tablet_manager.cc:1367] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.223s user 0.010s sys 0.010s
I20240501 14:01:09.393517 3316 raft_consensus.cc:348] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.393765 3316 raft_consensus.cc:374] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.393807 3319 tablet_bootstrap.cc:654] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.393831 3316 raft_consensus.cc:729] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.394117 3316 consensus_queue.cc:260] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.394631 3316 ts_tablet_manager.cc:1398] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:09.394747 3333 ts_tablet_manager.cc:1875] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:09.395155 3318 tablet_bootstrap.cc:654] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.397454 3321 raft_consensus.cc:348] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.397702 3321 raft_consensus.cc:374] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.397787 3321 raft_consensus.cc:729] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.398029 3321 consensus_queue.cc:260] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.401489 3320 ts_tablet_manager.cc:1398] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.058s user 0.000s sys 0.002s
I20240501 14:01:09.404479 3321 ts_tablet_manager.cc:1398] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240501 14:01:09.406679 3333 ts_tablet_manager.cc:1888] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:09.410684 3333 log.cc:1196] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/0c8060687b634344a68df56fc61acbfe
I20240501 14:01:09.410871 3320 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:09.411437 3333 ts_tablet_manager.cc:1909] T 0c8060687b634344a68df56fc61acbfe P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:09.413018 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table_new_name [id=53379e4c938649d38a2f40cb6aa2ab61]) successfully deleted
I20240501 14:01:09.414808 3320 tablet_bootstrap.cc:654] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:09.418682 3139 raft_consensus.cc:482] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.419035 3326 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:09.418889 3139 raft_consensus.cc:504] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.419164 3326 ts_tablet_manager.cc:1367] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.000s
I20240501 14:01:09.419688 3139 leader_election.cc:288] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:09.419867 3326 raft_consensus.cc:348] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.420044 3326 raft_consensus.cc:374] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.420094 3326 raft_consensus.cc:729] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.420449 3326 consensus_queue.cc:260] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.421240 3326 ts_tablet_manager.cc:1398] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:09.422029 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:09.422510 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:09.422720 29373 raft_consensus.cc:2454] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 0.
W20240501 14:01:09.424058 32163 leader_election.cc:341] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:09.424507 32163 leader_election.cc:302] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:09.426609 3139 raft_consensus.cc:2781] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:09.426728 3139 raft_consensus.cc:482] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:09.426841 3139 raft_consensus.cc:3037] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.430400 3139 raft_consensus.cc:504] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.432276 3139 leader_election.cc:288] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:09.432291 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:09.432951 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:09.433204 29377 raft_consensus.cc:3037] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:09.433231 32163 leader_election.cc:341] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:09.437245 29377 raft_consensus.cc:2454] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:01:09.437919 32163 leader_election.cc:302] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:09.438195 3139 raft_consensus.cc:2781] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:09.439092 3139 raft_consensus.cc:686] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:09.439507 3139 consensus_queue.cc:237] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.441303 32206 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:09.445003 3319 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:09.445185 3319 ts_tablet_manager.cc:1367] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.001s
I20240501 14:01:09.446138 3319 raft_consensus.cc:348] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.446403 3319 raft_consensus.cc:374] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.446491 3319 raft_consensus.cc:729] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.447260 3319 consensus_queue.cc:260] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.448002 3319 ts_tablet_manager.cc:1398] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:09.449246 32296 leader_election.cc:302] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:09.449851 2840 raft_consensus.cc:2781] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:09.450003 2840 raft_consensus.cc:686] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [term 2 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:09.450285 2840 consensus_queue.cc:237] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.455817 3090 consensus_queue.cc:1035] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.454569 29103 catalog_manager.cc:5484] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 1 to 2, leader changed from 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 2 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:09.468405 3295 consensus_queue.cc:1035] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.500047 29103 catalog_manager.cc:5484] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:09.504315 3320 tablet_bootstrap.cc:492] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:09.504830 3320 ts_tablet_manager.cc:1367] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.005s
I20240501 14:01:09.506616 3320 raft_consensus.cc:348] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.513823 3320 raft_consensus.cc:374] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.523137 3320 raft_consensus.cc:729] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.525267 3331 log.cc:1196] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/0c8060687b634344a68df56fc61acbfe
I20240501 14:01:09.525830 3331 ts_tablet_manager.cc:1909] T 0c8060687b634344a68df56fc61acbfe P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:01:09.526647 3320 consensus_queue.cc:260] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.527053 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 0c8060687b634344a68df56fc61acbfe (table rebalancer_dynamic_table_new_name [id=53379e4c938649d38a2f40cb6aa2ab61]) successfully deleted
I20240501 14:01:09.529640 3295 raft_consensus.cc:482] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.529812 3295 raft_consensus.cc:504] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.530942 3295 leader_election.cc:288] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:09.531471 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ae82d0e6848b2baf967306b1a068d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:01:09.531669 29635 raft_consensus.cc:2454] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:09.531796 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ae82d0e6848b2baf967306b1a068d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:09.531982 29506 raft_consensus.cc:2454] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 0.
I20240501 14:01:09.532244 31898 leader_election.cc:302] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:09.532671 3295 raft_consensus.cc:2781] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:09.532764 3295 raft_consensus.cc:482] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:09.533406 3295 raft_consensus.cc:3037] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.534657 29224 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:09.536239 3295 raft_consensus.cc:504] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.540349 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ae82d0e6848b2baf967306b1a068d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:09.540519 29635 raft_consensus.cc:3037] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.543198 29635 raft_consensus.cc:2454] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:09.543788 31898 leader_election.cc:302] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:09.544083 3311 raft_consensus.cc:2781] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:09.544221 3311 raft_consensus.cc:686] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:09.544533 3311 consensus_queue.cc:237] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:09.544741 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ae82d0e6848b2baf967306b1a068d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:09.544943 29506 raft_consensus.cc:3037] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.548635 29506 raft_consensus.cc:2454] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:01:09.549219 3295 leader_election.cc:288] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:09.553725 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:09.577847 3318 tablet_bootstrap.cc:492] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:09.577981 3320 ts_tablet_manager.cc:1398] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.073s user 0.001s sys 0.001s
I20240501 14:01:09.578794 3318 ts_tablet_manager.cc:1367] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.353s user 0.010s sys 0.048s
I20240501 14:01:09.578833 29105 catalog_manager.cc:5484] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:09.580144 3318 raft_consensus.cc:348] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.580418 3318 raft_consensus.cc:374] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:09.580557 3318 raft_consensus.cc:729] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:09.580847 3318 consensus_queue.cc:260] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.585072 3318 ts_tablet_manager.cc:1398] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:09.628582 3090 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.628782 3090 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:09.629518 3090 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:09.631044 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:09.631440 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:09.631667 29244 raft_consensus.cc:2379] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9903ed35bca144bcb74e9bf5102117e7 in current term 1: Already voted for candidate 522f4ff5e9084cbf8462196533c7d7aa in this term.
I20240501 14:01:09.632899 31899 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:09.633445 3090 raft_consensus.cc:3037] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.636469 3090 raft_consensus.cc:2726] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:09.687373 2632 raft_consensus.cc:482] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:09.687563 2632 raft_consensus.cc:504] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:09.688186 2632 leader_election.cc:288] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:09.688935 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:09.692960 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05d6d2f1f1140c7bea1e084574b51fe" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:09.693207 29377 raft_consensus.cc:2379] T c05d6d2f1f1140c7bea1e084574b51fe P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a697f3a1ee544daa7073814ab845d58 in current term 1: Already voted for candidate 6a446ac802a7457d9f27076781f21d79 in this term.
I20240501 14:01:09.693701 32032 leader_election.cc:302] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35
I20240501 14:01:09.694526 2632 raft_consensus.cc:3037] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:09.696821 2632 raft_consensus.cc:2726] T c05d6d2f1f1140c7bea1e084574b51fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:09.699054 2856 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.711766 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:09.714320 3335 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.718645 29637 raft_consensus.cc:1261] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:09.719137 29507 raft_consensus.cc:1261] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:09.719921 3090 consensus_queue.cc:1035] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.721274 3090 consensus_queue.cc:1035] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.735881 29105 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]). Expected version 1 got 0
I20240501 14:01:09.738725 3336 consensus_queue.cc:1035] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.740437 3364 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:09.746039 3364 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:09.755079 3365 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:09.757601 3365 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:09.758773 29105 catalog_manager.cc:5576] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947) does not have the latest schema for tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]). Expected version 1 got 0
I20240501 14:01:09.765847 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce] alter complete (version 1)
I20240501 14:01:09.814633 3193 consensus_queue.cc:1035] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.828563 29105 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]). Expected version 1 got 0
I20240501 14:01:09.832924 3139 consensus_queue.cc:1035] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.845571 32227 raft_consensus.cc:1261] T d2cf9b7f68034fcb80cf620687ca4a81 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:09.846151 3234 consensus_queue.cc:1035] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:09.867323 31964 raft_consensus.cc:1261] T d2cf9b7f68034fcb80cf620687ca4a81 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:09.870815 2840 consensus_queue.cc:1035] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:09.877652 3374 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:09.878580 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:09.881717 29100 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]). Expected version 1 got 0
I20240501 14:01:09.882349 3374 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:09.895992 3364 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:09.896538 3364 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:09.897007 3365 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:09.897514 3365 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:09.907820 3374 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:09.908490 3374 tablet.cc:1620] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: MemRowSet was empty: no flush needed.
I20240501 14:01:09.909734 29100 catalog_manager.cc:5576] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187) does not have the latest schema for tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce]). Expected version 2 got 1
I20240501 14:01:09.916697 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=5faa3a1faf91418ca28c1fe3d30553ce] alter complete (version 2)
I20240501 14:01:09.937283 3379 consensus_queue.cc:1035] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:01:09.950630 29377 consensus_queue.cc:237] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:09.954313 32227 raft_consensus.cc:1261] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:09.954813 29246 raft_consensus.cc:1261] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:09.955776 3237 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:09.957580 3237 consensus_queue.cc:1035] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:09.989915 3236 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:09.995453 29103 catalog_manager.cc:5484] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:09.999522 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e94a79c4514fe4a3ae8e41c7792dc9 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:10.000231 3331 tablet_replica.cc:330] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:10.000437 3331 raft_consensus.cc:2229] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:10.000741 3331 raft_consensus.cc:2258] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.001528 3331 ts_tablet_manager.cc:1875] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:10.008261 3331 ts_tablet_manager.cc:1888] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:10.008522 3331 log.cc:1196] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6e94a79c4514fe4a3ae8e41c7792dc9
I20240501 14:01:09.997753 29246 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:10.009697 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet a6e94a79c4514fe4a3ae8e41c7792dc9 (table rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]) successfully deleted
I20240501 14:01:10.015874 32227 raft_consensus.cc:2932] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:10.019567 29103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:10.031931 3364 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:10.034816 3365 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:10.039666 3374 tablet.cc:1722] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:10.049844 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=5faa3a1faf91418ca28c1fe3d30553ce] alter complete (version 3)
I20240501 14:01:10.053035 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:10.064770 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.065100 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.065275 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.065477 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.070365 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.070780 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.071480 3318 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:10.072518 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7cd18adcd844f4be7ddd343cda3df4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.072914 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7cd18adcd844f4be7ddd343cda3df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.073112 3318 tablet_bootstrap.cc:654] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.074585 32077 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7cd18adcd844f4be7ddd343cda3df4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.075004 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7cd18adcd844f4be7ddd343cda3df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.079732 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7cd18adcd844f4be7ddd343cda3df4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.080258 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7cd18adcd844f4be7ddd343cda3df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.090054 32076 tablet_service.cc:1467] Processing CreateTablet for tablet e190940ae8b44ca2bf1ee496a9d624ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.090483 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190940ae8b44ca2bf1ee496a9d624ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.091498 29356 tablet_service.cc:1467] Processing CreateTablet for tablet e190940ae8b44ca2bf1ee496a9d624ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.091929 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190940ae8b44ca2bf1ee496a9d624ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.093056 29486 tablet_service.cc:1467] Processing CreateTablet for tablet e190940ae8b44ca2bf1ee496a9d624ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.093454 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190940ae8b44ca2bf1ee496a9d624ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.103986 3398 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:10.106694 3398 tablet_bootstrap.cc:654] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.110225 3318 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:10.110551 3318 ts_tablet_manager.cc:1367] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.000s
I20240501 14:01:10.111541 3318 raft_consensus.cc:348] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.111894 3318 raft_consensus.cc:374] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.111986 3318 raft_consensus.cc:729] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.113324 3318 consensus_queue.cc:260] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.114106 3318 ts_tablet_manager.cc:1398] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:10.116936 3318 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:10.119165 3318 tablet_bootstrap.cc:654] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.119671 3399 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:10.121676 3399 tablet_bootstrap.cc:654] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.123175 3400 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:10.125124 3398 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:10.125324 3398 ts_tablet_manager.cc:1367] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240501 14:01:10.126041 3318 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:10.126220 3318 ts_tablet_manager.cc:1367] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:01:10.126224 3398 raft_consensus.cc:348] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.126480 3398 raft_consensus.cc:374] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.126572 3398 raft_consensus.cc:729] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.126864 3398 consensus_queue.cc:260] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.127085 3400 tablet_bootstrap.cc:654] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.127194 3318 raft_consensus.cc:348] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.127451 3318 raft_consensus.cc:374] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.127570 3318 raft_consensus.cc:729] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.127704 3398 ts_tablet_manager.cc:1398] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:10.128069 3398 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:10.128155 3318 consensus_queue.cc:260] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.129465 3398 tablet_bootstrap.cc:654] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.135926 3318 ts_tablet_manager.cc:1398] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 14:01:10.152244 3318 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:10.152810 29105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:10.153082 29105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:10.156572 29105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet 0e9ae82d0e6848b2baf967306b1a068d
I20240501 14:01:10.157431 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:10 UTC) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:10.157680 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:10 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:10.157744 3331 tablet_replica.cc:330] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:10.157938 3331 raft_consensus.cc:2229] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:10.158114 3331 raft_consensus.cc:2258] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.158457 3318 tablet_bootstrap.cc:654] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.158664 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ae82d0e6848b2baf967306b1a068d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:10.159570 3331 ts_tablet_manager.cc:1875] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:10.161159 3413 tablet_replica.cc:330] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:10.161422 3413 raft_consensus.cc:2229] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:10.161829 3413 raft_consensus.cc:2258] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.163252 3413 ts_tablet_manager.cc:1875] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:10.179664 3413 ts_tablet_manager.cc:1888] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:10.182617 3413 log.cc:1196] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/0e9ae82d0e6848b2baf967306b1a068d
I20240501 14:01:10.183472 3413 ts_tablet_manager.cc:1909] T 0e9ae82d0e6848b2baf967306b1a068d P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:10.185956 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table_new_name [id=5faa3a1faf91418ca28c1fe3d30553ce]) successfully deleted
I20240501 14:01:10.197914 3399 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:10.198297 3399 ts_tablet_manager.cc:1367] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.001s
I20240501 14:01:10.199417 3399 raft_consensus.cc:348] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.203141 3399 raft_consensus.cc:374] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.203402 3399 raft_consensus.cc:729] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.203903 3399 consensus_queue.cc:260] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.207440 3331 ts_tablet_manager.cc:1888] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:10.207564 3331 log.cc:1196] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e9ae82d0e6848b2baf967306b1a068d
I20240501 14:01:10.207942 3331 ts_tablet_manager.cc:1909] T 0e9ae82d0e6848b2baf967306b1a068d P 522f4ff5e9084cbf8462196533c7d7aa: Deleting consensus metadata
I20240501 14:01:10.209126 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table_new_name [id=5faa3a1faf91418ca28c1fe3d30553ce]) successfully deleted
I20240501 14:01:10.211560 3399 ts_tablet_manager.cc:1398] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 14:01:10.213018 3400 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:10.218627 3399 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:10.221289 3400 ts_tablet_manager.cc:1367] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.002s
I20240501 14:01:10.228457 3400 raft_consensus.cc:348] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.231279 3399 tablet_bootstrap.cc:654] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.234469 3400 raft_consensus.cc:374] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.245185 3400 raft_consensus.cc:729] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.246352 3400 consensus_queue.cc:260] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.254272 3400 ts_tablet_manager.cc:1398] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
I20240501 14:01:10.254781 3412 tablet_replica.cc:330] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:10.256306 3399 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:10.256599 3412 raft_consensus.cc:2229] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:10.256834 3412 raft_consensus.cc:2258] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.257072 3399 ts_tablet_manager.cc:1367] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.002s
I20240501 14:01:10.257988 3412 ts_tablet_manager.cc:1875] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:10.260226 29105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:10.269927 32206 tablet_service.cc:1467] Processing CreateTablet for tablet f91b29db70ce406db588a89d6c2f34f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]), partition=
I20240501 14:01:10.270368 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91b29db70ce406db588a89d6c2f34f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.273365 3389 raft_consensus.cc:482] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:10.273451 32076 tablet_service.cc:1467] Processing CreateTablet for tablet f91b29db70ce406db588a89d6c2f34f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]), partition=
I20240501 14:01:10.273626 3389 raft_consensus.cc:504] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.273993 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91b29db70ce406db588a89d6c2f34f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.274755 3389 leader_election.cc:288] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:10.275812 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a38ce68ec4d4ba3a813bdb0ec314d4d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:10.292711 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a38ce68ec4d4ba3a813bdb0ec314d4d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:10.293062 29373 raft_consensus.cc:2454] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:10.295534 3416 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:10.295753 32032 leader_election.cc:302] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:10.296631 2632 raft_consensus.cc:2781] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:10.296773 2632 raft_consensus.cc:482] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:10.296901 2632 raft_consensus.cc:3037] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:10.297178 32031 leader_election.cc:341] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:10.299593 3416 tablet_bootstrap.cc:654] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.300416 3398 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:10.300551 3398 ts_tablet_manager.cc:1367] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.173s user 0.051s sys 0.041s
I20240501 14:01:10.300734 3399 raft_consensus.cc:348] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.301505 3398 raft_consensus.cc:348] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.301759 3398 raft_consensus.cc:374] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.301889 3398 raft_consensus.cc:729] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.301076 3399 raft_consensus.cc:374] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.301971 3399 raft_consensus.cc:729] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.302258 3398 consensus_queue.cc:260] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.303458 3398 ts_tablet_manager.cc:1398] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:10.302301 3399 consensus_queue.cc:260] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.304158 3399 ts_tablet_manager.cc:1398] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240501 14:01:10.307395 3416 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:10.308095 3416 ts_tablet_manager.cc:1367] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240501 14:01:10.308615 3399 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:10.309177 3416 raft_consensus.cc:348] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.309413 3416 raft_consensus.cc:374] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.309512 3416 raft_consensus.cc:729] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.310249 3416 consensus_queue.cc:260] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.310830 3416 ts_tablet_manager.cc:1398] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:10.300127 2632 raft_consensus.cc:504] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.315339 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a38ce68ec4d4ba3a813bdb0ec314d4d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:10.315541 29507 raft_consensus.cc:3037] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.316020 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a38ce68ec4d4ba3a813bdb0ec314d4d" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:10.316185 29377 raft_consensus.cc:3037] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.318042 29507 raft_consensus.cc:2454] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:10.318360 2632 leader_election.cc:288] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:10.318552 32031 leader_election.cc:302] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa; no voters:
I20240501 14:01:10.318853 2632 raft_consensus.cc:2781] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:10.319167 2632 raft_consensus.cc:686] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:10.319496 2632 consensus_queue.cc:237] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:10.320780 29377 raft_consensus.cc:2454] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:10.321410 29617 tablet_service.cc:1467] Processing CreateTablet for tablet f91b29db70ce406db588a89d6c2f34f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]), partition=
I20240501 14:01:10.322309 3399 tablet_bootstrap.cc:654] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.322376 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91b29db70ce406db588a89d6c2f34f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.328020 3412 ts_tablet_manager.cc:1888] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:10.328153 3412 log.cc:1196] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e9ae82d0e6848b2baf967306b1a068d
I20240501 14:01:10.328442 3412 ts_tablet_manager.cc:1909] T 0e9ae82d0e6848b2baf967306b1a068d P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:10.329746 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 0e9ae82d0e6848b2baf967306b1a068d (table rebalancer_dynamic_table_new_name [id=5faa3a1faf91418ca28c1fe3d30553ce]) successfully deleted
I20240501 14:01:10.312134 3318 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:10.330232 3318 ts_tablet_manager.cc:1367] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.178s user 0.035s sys 0.093s
I20240501 14:01:10.333400 3318 raft_consensus.cc:348] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.333827 3318 raft_consensus.cc:374] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.333963 3318 raft_consensus.cc:729] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.334568 3318 consensus_queue.cc:260] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.335739 3318 ts_tablet_manager.cc:1398] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:01:10.336076 3318 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:10.336822 3399 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:10.337733 3318 tablet_bootstrap.cc:654] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.337056 3399 ts_tablet_manager.cc:1367] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240501 14:01:10.339747 3399 raft_consensus.cc:348] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.341985 3399 raft_consensus.cc:374] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.342124 3399 raft_consensus.cc:729] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.342530 3399 consensus_queue.cc:260] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.343367 3399 ts_tablet_manager.cc:1398] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:10.344060 3420 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:10.349721 3420 tablet_bootstrap.cc:654] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:10.350993 3318 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:10.351152 3318 ts_tablet_manager.cc:1367] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20240501 14:01:10.351083 29103 catalog_manager.cc:5484] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:10.352059 3318 raft_consensus.cc:348] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.352255 3318 raft_consensus.cc:374] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.352546 3318 raft_consensus.cc:729] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.352785 3318 consensus_queue.cc:260] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.353377 3318 ts_tablet_manager.cc:1398] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:10.362668 3420 tablet_bootstrap.cc:492] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:10.362893 3420 ts_tablet_manager.cc:1367] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20240501 14:01:10.363619 3420 raft_consensus.cc:348] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.363775 3420 raft_consensus.cc:374] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:10.363842 3420 raft_consensus.cc:729] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:10.364015 3420 consensus_queue.cc:260] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.364518 3420 ts_tablet_manager.cc:1398] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:10.382548 3389 raft_consensus.cc:482] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:10.382782 3389 raft_consensus.cc:504] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.383412 3389 leader_election.cc:288] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:10.384009 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7cd18adcd844f4be7ddd343cda3df4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:10.384258 32361 raft_consensus.cc:2454] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:10.384577 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7cd18adcd844f4be7ddd343cda3df4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:10.384749 32031 leader_election.cc:302] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:10.384786 29505 raft_consensus.cc:2454] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 0.
I20240501 14:01:10.385129 3389 raft_consensus.cc:2781] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:10.385217 3389 raft_consensus.cc:482] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:10.385313 3389 raft_consensus.cc:3037] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.393114 3389 raft_consensus.cc:504] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.395138 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7cd18adcd844f4be7ddd343cda3df4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:10.395340 29505 raft_consensus.cc:3037] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.396582 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7cd18adcd844f4be7ddd343cda3df4" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:10.396798 32361 raft_consensus.cc:3037] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.398916 29505 raft_consensus.cc:2454] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:10.399644 32361 raft_consensus.cc:2454] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:01:10.400297 32031 leader_election.cc:302] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:10.400625 3341 raft_consensus.cc:2781] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:10.402472 3389 leader_election.cc:288] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:10.401675 3336 raft_consensus.cc:482] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:10.403069 3336 raft_consensus.cc:504] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.403889 3336 leader_election.cc:288] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:10.404440 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:10.404707 32096 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:01:10.404697 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:10.404937 29377 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 0.
I20240501 14:01:10.405298 29442 leader_election.cc:302] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa; no voters:
I20240501 14:01:10.406157 3336 raft_consensus.cc:2781] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:10.406255 3336 raft_consensus.cc:482] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:10.406332 3336 raft_consensus.cc:3037] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.410831 3336 raft_consensus.cc:504] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.412431 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:10.412658 32096 raft_consensus.cc:3037] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.413015 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:10.413271 29377 raft_consensus.cc:3037] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.414680 3341 raft_consensus.cc:686] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:10.415036 3341 consensus_queue.cc:237] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:10.415534 32096 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:01:10.416154 29442 leader_election.cc:302] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa; no voters:
I20240501 14:01:10.416235 29377 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 1.
I20240501 14:01:10.416440 3193 raft_consensus.cc:2781] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:10.417388 3336 leader_election.cc:288] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:10.428237 29096 catalog_manager.cc:5484] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:10.448438 3335 raft_consensus.cc:482] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:10.448621 3335 raft_consensus.cc:504] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.449316 3335 leader_election.cc:288] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:10.450278 3193 raft_consensus.cc:686] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:01:10.450629 3193 consensus_queue.cc:237] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:10.450815 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91b29db70ce406db588a89d6c2f34f6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:10.450913 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91b29db70ce406db588a89d6c2f34f6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:10.451087 32227 raft_consensus.cc:2454] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:10.451135 32095 raft_consensus.cc:2454] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:10.451575 29571 leader_election.cc:302] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:10.452804 3335 raft_consensus.cc:2781] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:10.452888 3335 raft_consensus.cc:482] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:10.452966 3335 raft_consensus.cc:3037] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.454669 29096 catalog_manager.cc:5484] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:10.455767 3335 raft_consensus.cc:504] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.457656 32227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91b29db70ce406db588a89d6c2f34f6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:10.457917 32227 raft_consensus.cc:3037] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.459959 32227 raft_consensus.cc:2454] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:10.460318 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91b29db70ce406db588a89d6c2f34f6" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:10.460557 32095 raft_consensus.cc:3037] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:10.461221 29571 leader_election.cc:302] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:10.461477 2856 raft_consensus.cc:2781] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:10.461506 3335 leader_election.cc:288] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:10.461642 2856 raft_consensus.cc:686] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:10.461908 2856 consensus_queue.cc:237] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:10.464133 32095 raft_consensus.cc:2454] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:10.468526 29096 catalog_manager.cc:5484] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:10.589637 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:10.671963 32227 raft_consensus.cc:1261] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.677033 2856 consensus_queue.cc:1035] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.678743 32095 raft_consensus.cc:1261] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.679502 2856 consensus_queue.cc:1035] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.725100 3466 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:10.725608 3466 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:10.727062 29103 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]). Expected version 1 got 0
I20240501 14:01:10.733095 29105 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]). Expected version 1 got 0
I20240501 14:01:10.733567 3341 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.737231 29097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16] alter complete (version 1)
I20240501 14:01:10.738152 29103 catalog_manager.cc:5576] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745) does not have the latest schema for tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]). Expected version 1 got 0
I20240501 14:01:10.740509 2632 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.753306 3468 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:10.754426 3468 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:10.837181 3470 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:10.837782 3470 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:10.842816 29104 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]). Expected version 1 got 0
I20240501 14:01:10.852368 29104 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:10.868503 3466 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:10.868719 29097 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16]). Expected version 2 got 1
I20240501 14:01:10.869035 3466 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:10.871436 3468 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:10.871923 3468 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: MemRowSet was empty: no flush needed.
I20240501 14:01:10.888821 29224 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:10.901448 3470 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:10.902011 3470 tablet.cc:1620] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:10.902640 29104 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=6830237f22b54d419263f76cce341c16] alter complete (version 2)
I20240501 14:01:10.943826 3341 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.966881 29097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:10.971025 3193 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:10.980528 32206 tablet_service.cc:1467] Processing CreateTablet for tablet 9320d3f885be46e4bf165243eaef41aa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.980836 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 9320d3f885be46e4bf165243eaef41aa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.981050 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9320d3f885be46e4bf165243eaef41aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.981339 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9320d3f885be46e4bf165243eaef41aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.982026 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 9320d3f885be46e4bf165243eaef41aa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.982466 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9320d3f885be46e4bf165243eaef41aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.985395 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0fcc86017b4589b1a960c078e7ec57 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.985786 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0fcc86017b4589b1a960c078e7ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.986316 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0fcc86017b4589b1a960c078e7ec57 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.986845 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0fcc86017b4589b1a960c078e7ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.986316 29486 tablet_service.cc:1467] Processing CreateTablet for tablet e83b7b4decde4e459e49de9eb05b8c0f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.989917 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83b7b4decde4e459e49de9eb05b8c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.990015 29226 tablet_service.cc:1467] Processing CreateTablet for tablet e83b7b4decde4e459e49de9eb05b8c0f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.990406 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83b7b4decde4e459e49de9eb05b8c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.994072 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0fcc86017b4589b1a960c078e7ec57 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.994509 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0fcc86017b4589b1a960c078e7ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.996263 32340 tablet_service.cc:1467] Processing CreateTablet for tablet e83b7b4decde4e459e49de9eb05b8c0f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:10.996621 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83b7b4decde4e459e49de9eb05b8c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.008641 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:11.014360 2632 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.014914 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:11.015002 3491 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:11.027432 3493 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:11.029386 3493 tablet_bootstrap.cc:654] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.032299 3491 tablet_bootstrap.cc:654] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.036964 3466 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:11.038462 3468 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:11.039384 3489 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:11.041872 3489 tablet_bootstrap.cc:654] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.063366 3193 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.072487 3489 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:11.073086 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:11.074986 3489 ts_tablet_manager.cc:1367] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.008s
I20240501 14:01:11.079560 3489 raft_consensus.cc:348] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.082309 3489 raft_consensus.cc:374] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.082862 3470 tablet.cc:1722] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:11.083197 3489 raft_consensus.cc:729] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.084460 3489 consensus_queue.cc:260] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.086875 3489 ts_tablet_manager.cc:1398] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 14:01:11.089248 3489 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:11.095086 3489 tablet_bootstrap.cc:654] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.098419 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=6830237f22b54d419263f76cce341c16] alter complete (version 3)
I20240501 14:01:11.103715 3492 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:11.105587 3492 tablet_bootstrap.cc:654] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.106979 3490 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:11.108534 3490 tablet_bootstrap.cc:654] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.112048 3489 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:11.112282 3489 ts_tablet_manager.cc:1367] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20240501 14:01:11.113117 3489 raft_consensus.cc:348] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.113399 3489 raft_consensus.cc:374] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.113507 3489 raft_consensus.cc:729] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.113806 3489 consensus_queue.cc:260] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.115759 3489 ts_tablet_manager.cc:1398] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:11.138324 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:11.143749 3490 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:11.144003 3490 ts_tablet_manager.cc:1367] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20240501 14:01:11.144778 3490 raft_consensus.cc:348] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.145102 3490 raft_consensus.cc:374] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.145211 3490 raft_consensus.cc:729] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.145463 3490 consensus_queue.cc:260] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.148702 3490 ts_tablet_manager.cc:1398] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:01:11.149626 3490 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:11.156808 3500 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:11.157869 3492 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:11.158057 3492 ts_tablet_manager.cc:1367] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240501 14:01:11.161279 3492 raft_consensus.cc:348] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.161682 2289 raft_consensus.cc:482] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:11.161809 2289 raft_consensus.cc:504] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.162870 2289 leader_election.cc:288] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:11.163841 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:01:11.164469 29180 leader_election.cc:341] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:11.164809 3493 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:11.165024 3492 raft_consensus.cc:374] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.165203 3493 ts_tablet_manager.cc:1367] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.014s
I20240501 14:01:11.165299 3492 raft_consensus.cc:729] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.166132 3493 raft_consensus.cc:348] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.169168 3492 consensus_queue.cc:260] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.170814 3490 tablet_bootstrap.cc:654] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.172781 3492 ts_tablet_manager.cc:1398] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:01:11.185559 3491 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:11.187059 3500 tablet_bootstrap.cc:654] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.188418 3493 raft_consensus.cc:374] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.188764 3491 ts_tablet_manager.cc:1367] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.174s user 0.001s sys 0.021s
I20240501 14:01:11.189941 3491 raft_consensus.cc:348] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.191013 3493 raft_consensus.cc:729] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.194437 3493 consensus_queue.cc:260] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.197741 3491 raft_consensus.cc:374] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.197870 3491 raft_consensus.cc:729] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.198132 3491 consensus_queue.cc:260] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.199472 3493 ts_tablet_manager.cc:1398] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240501 14:01:11.201154 3493 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:11.174697 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:11.211351 3491 ts_tablet_manager.cc:1398] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:01:11.212108 29617 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:11.220299 3493 tablet_bootstrap.cc:654] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.225188 32342 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:11.228171 29097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:11.230206 29097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:11.232008 3493 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:11.232847 3493 ts_tablet_manager.cc:1367] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.002s
I20240501 14:01:11.233951 3493 raft_consensus.cc:348] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.236987 3493 raft_consensus.cc:374] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.237320 3493 raft_consensus.cc:729] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
W20240501 14:01:11.238648 29180 leader_election.cc:341] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:11.238772 29180 leader_election.cc:302] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:11.239135 2289 raft_consensus.cc:2726] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:11.239491 3493 consensus_queue.cc:260] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.246965 3493 ts_tablet_manager.cc:1398] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:11.256239 29357 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:11.259478 29097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet f91b29db70ce406db588a89d6c2f34f6
I20240501 14:01:11.260318 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet f91b29db70ce406db588a89d6c2f34f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:11 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:11.261224 3234 raft_consensus.cc:482] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:11.261440 3234 raft_consensus.cc:504] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.261773 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet f91b29db70ce406db588a89d6c2f34f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:11 UTC) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:11.262434 3234 leader_election.cc:288] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:11.263439 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:11.263751 29244 raft_consensus.cc:2454] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:11.264429 32294 leader_election.cc:302] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:11.265097 3234 raft_consensus.cc:2781] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:11.265225 3234 raft_consensus.cc:482] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:11.265358 3234 raft_consensus.cc:3037] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.273100 3234 raft_consensus.cc:504] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.275054 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:11.275307 29244 raft_consensus.cc:3037] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.277987 29244 raft_consensus.cc:2454] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:11.278569 32294 leader_election.cc:302] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:01:11.278894 2840 raft_consensus.cc:2781] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:11.285204 2289 raft_consensus.cc:482] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:11.285336 2289 raft_consensus.cc:504] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.286053 2289 leader_election.cc:288] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:11.287187 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9320d3f885be46e4bf165243eaef41aa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:11.287359 3500 tablet_bootstrap.cc:492] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
W20240501 14:01:11.287950 29181 leader_election.cc:341] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:11.287794 3500 ts_tablet_manager.cc:1367] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.000s
I20240501 14:01:11.288914 2840 raft_consensus.cc:686] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:11.289363 2840 consensus_queue.cc:237] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.289615 3500 raft_consensus.cc:348] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.290597 3234 leader_election.cc:288] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:11.291020 3500 raft_consensus.cc:374] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.291326 3500 raft_consensus.cc:729] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.291824 3500 consensus_queue.cc:260] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.292932 3500 ts_tablet_manager.cc:1398] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:11.299942 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet f91b29db70ce406db588a89d6c2f34f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:11 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:11.314975 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:11.344398 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9320d3f885be46e4bf165243eaef41aa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:11.344796 32224 raft_consensus.cc:2454] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:11.346297 29180 leader_election.cc:302] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters: 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:11.347187 3230 raft_consensus.cc:2781] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:11.347275 3230 raft_consensus.cc:482] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:11.347348 3230 raft_consensus.cc:3037] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.350466 3230 raft_consensus.cc:504] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.351143 3230 leader_election.cc:288] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:11.351563 3520 tablet_replica.cc:330] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:11.352236 3520 raft_consensus.cc:2229] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.352429 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9320d3f885be46e4bf165243eaef41aa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:11.352646 32224 raft_consensus.cc:3037] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.353775 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9320d3f885be46e4bf165243eaef41aa" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:11.353987 31965 raft_consensus.cc:3037] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.355726 32224 raft_consensus.cc:2454] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:11.352491 3520 raft_consensus.cc:2258] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.358418 31965 raft_consensus.cc:2454] T 9320d3f885be46e4bf165243eaef41aa P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:11.358647 3520 ts_tablet_manager.cc:1875] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:11.358713 29180 leader_election.cc:302] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:11.359009 2289 raft_consensus.cc:2781] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:11.359129 2289 raft_consensus.cc:686] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:11.359372 2289 consensus_queue.cc:237] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:11.381451 3520 ts_tablet_manager.cc:1888] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:11.381726 3520 log.cc:1196] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/f91b29db70ce406db588a89d6c2f34f6
I20240501 14:01:11.382611 3520 ts_tablet_manager.cc:1909] T f91b29db70ce406db588a89d6c2f34f6 P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:11.384126 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:11.384819 29505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83b7b4decde4e459e49de9eb05b8c0f" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:11.385221 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table_new_name [id=6830237f22b54d419263f76cce341c16]) successfully deleted
W20240501 14:01:11.387619 32295 leader_election.cc:341] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:01:11.388046 32295 leader_election.cc:341] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:11.403450 3516 tablet_replica.cc:330] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:11.404865 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:11.408249 3516 raft_consensus.cc:2229] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.408571 3516 raft_consensus.cc:2258] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.410805 3516 ts_tablet_manager.cc:1875] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:11.414686 29616 tablet_service.cc:1467] Processing CreateTablet for tablet a899520b90754b0e886db727ad89574a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]), partition=
I20240501 14:01:11.414907 32207 tablet_service.cc:1467] Processing CreateTablet for tablet a899520b90754b0e886db727ad89574a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]), partition=
I20240501 14:01:11.415087 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a899520b90754b0e886db727ad89574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.415346 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a899520b90754b0e886db727ad89574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.421067 3492 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:11.421660 31944 tablet_service.cc:1467] Processing CreateTablet for tablet a899520b90754b0e886db727ad89574a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]), partition=
I20240501 14:01:11.422049 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a899520b90754b0e886db727ad89574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.423188 3492 tablet_bootstrap.cc:654] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.433506 3498 raft_consensus.cc:482] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:11.433657 3498 raft_consensus.cc:504] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.434489 3498 leader_election.cc:288] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:01:11.436156 29572 consensus_peers.cc:485] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:11.436211 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0fcc86017b4589b1a960c078e7ec57" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa" is_pre_election: true
I20240501 14:01:11.436434 29507 raft_consensus.cc:2454] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:11.437806 29571 leader_election.cc:302] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:11.438155 3498 raft_consensus.cc:2781] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:11.438289 3498 raft_consensus.cc:482] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:11.438434 3498 raft_consensus.cc:3037] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.441447 3498 raft_consensus.cc:504] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.442559 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0fcc86017b4589b1a960c078e7ec57" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:11.442477 29096 catalog_manager.cc:5484] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:11.442749 29507 raft_consensus.cc:3037] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.443130 3515 tablet_replica.cc:330] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:11.443344 3515 raft_consensus.cc:2229] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:11.443693 3515 raft_consensus.cc:2258] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.444712 3515 ts_tablet_manager.cc:1875] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:11.445479 3498 leader_election.cc:288] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:11.446552 29507 raft_consensus.cc:2454] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:11.447110 29571 leader_election.cc:302] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:11.447528 3498 raft_consensus.cc:2781] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:11.447700 3498 raft_consensus.cc:686] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:11.448002 3498 consensus_queue.cc:237] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:11.450651 3500 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:11.452754 3500 tablet_bootstrap.cc:654] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
W20240501 14:01:11.452817 32161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 155269us
I20240501 14:01:11.454066 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0fcc86017b4589b1a960c078e7ec57" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:01:11.454288 32361 raft_consensus.cc:2454] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:11.454783 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0fcc86017b4589b1a960c078e7ec57" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:11.454965 32361 raft_consensus.cc:3037] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.457649 32361 raft_consensus.cc:2454] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:11.461884 3491 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:11.463845 3492 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:11.464063 3492 ts_tablet_manager.cc:1367] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.003s
I20240501 14:01:11.464316 3491 tablet_bootstrap.cc:654] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:11.465306 3492 raft_consensus.cc:348] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.465687 3492 raft_consensus.cc:374] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.465811 3492 raft_consensus.cc:729] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.465858 3490 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: No bootstrap required, opened a new log
I20240501 14:01:11.466161 3490 ts_tablet_manager.cc:1367] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.317s user 0.011s sys 0.025s
I20240501 14:01:11.466212 3492 consensus_queue.cc:260] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.466922 3492 ts_tablet_manager.cc:1398] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:11.467046 3490 raft_consensus.cc:348] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.467658 3490 raft_consensus.cc:374] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.467983 3490 raft_consensus.cc:729] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.479928 29096 catalog_manager.cc:5484] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
W20240501 14:01:11.485954 29181 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 100029us
I20240501 14:01:11.487977 29097 catalog_manager.cc:5484] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:11.488581 3491 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:11.488754 3491 ts_tablet_manager.cc:1367] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
W20240501 14:01:11.498481 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 52539.9us
I20240501 14:01:11.501852 3491 raft_consensus.cc:348] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.502100 3491 raft_consensus.cc:374] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.502189 3491 raft_consensus.cc:729] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.503090 3491 consensus_queue.cc:260] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.504638 3491 ts_tablet_manager.cc:1398] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 14:01:11.507802 3515 ts_tablet_manager.cc:1888] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:11.507939 3515 log.cc:1196] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/f91b29db70ce406db588a89d6c2f34f6
I20240501 14:01:11.525251 3500 tablet_bootstrap.cc:492] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:11.525578 3500 ts_tablet_manager.cc:1367] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.005s
I20240501 14:01:11.526262 3500 raft_consensus.cc:348] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.526463 3500 raft_consensus.cc:374] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:11.526538 3500 raft_consensus.cc:729] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:11.526847 3500 consensus_queue.cc:260] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.527333 3500 ts_tablet_manager.cc:1398] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 14:01:11.521270 32296 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 146978us
I20240501 14:01:11.534540 3490 consensus_queue.cc:260] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:11.535785 3490 ts_tablet_manager.cc:1398] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.069s user 0.000s sys 0.001s
W20240501 14:01:11.551295 29181 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 49581.8us
I20240501 14:01:11.551905 3516 ts_tablet_manager.cc:1888] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:11.552209 3516 log.cc:1196] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/f91b29db70ce406db588a89d6c2f34f6
I20240501 14:01:11.552762 3516 ts_tablet_manager.cc:1909] T f91b29db70ce406db588a89d6c2f34f6 P 0a697f3a1ee544daa7073814ab845d58: Deleting consensus metadata
I20240501 14:01:11.562336 3498 raft_consensus.cc:482] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:11.562496 3498 raft_consensus.cc:504] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.563529 3498 leader_election.cc:288] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:11.563678 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a899520b90754b0e886db727ad89574a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:11.563911 31965 raft_consensus.cc:2454] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:11.564280 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a899520b90754b0e886db727ad89574a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:11.564509 32224 raft_consensus.cc:2454] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:11.564939 29571 leader_election.cc:302] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:11.565227 3498 raft_consensus.cc:2781] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:11.565322 3498 raft_consensus.cc:482] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:11.565403 3498 raft_consensus.cc:3037] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.568037 3498 raft_consensus.cc:504] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.568902 3498 leader_election.cc:288] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:11.569015 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a899520b90754b0e886db727ad89574a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:11.569015 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a899520b90754b0e886db727ad89574a" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:11.569198 31965 raft_consensus.cc:3037] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.569234 32224 raft_consensus.cc:3037] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.572307 31965 raft_consensus.cc:2454] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:11.572346 32224 raft_consensus.cc:2454] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:11.572818 29571 leader_election.cc:302] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:11.573132 3498 raft_consensus.cc:2781] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:11.574280 3498 raft_consensus.cc:686] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:11.574681 3498 consensus_queue.cc:237] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:11.573630 3515 ts_tablet_manager.cc:1909] T f91b29db70ce406db588a89d6c2f34f6 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:11.584059 29097 catalog_manager.cc:5484] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:11.585166 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table_new_name [id=6830237f22b54d419263f76cce341c16]) successfully deleted
I20240501 14:01:11.586108 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet f91b29db70ce406db588a89d6c2f34f6 (table rebalancer_dynamic_table_new_name [id=6830237f22b54d419263f76cce341c16]) successfully deleted
I20240501 14:01:11.682152 29635 consensus_queue.cc:237] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:11.684559 29377 raft_consensus.cc:1261] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.684576 32361 raft_consensus.cc:1261] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.685045 3498 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.685290 2856 consensus_queue.cc:1035] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.688187 2856 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:11.688767 29377 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:11.689270 32361 raft_consensus.cc:2932] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:11.694713 29097 catalog_manager.cc:5484] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:11.700474 3521 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.701514 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e2a1f3418c477aa65b77bfa171a5a1 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:11.701864 3516 tablet_replica.cc:330] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:11.702106 3516 raft_consensus.cc:2229] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.702355 3516 raft_consensus.cc:2258] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.703055 3516 ts_tablet_manager.cc:1875] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:11.720206 29507 raft_consensus.cc:3037] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:11.721177 3516 ts_tablet_manager.cc:1888] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:11.721469 3516 log.cc:1196] T e5e2a1f3418c477aa65b77bfa171a5a1 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/e5e2a1f3418c477aa65b77bfa171a5a1
I20240501 14:01:11.722406 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet e5e2a1f3418c477aa65b77bfa171a5a1 (table rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]) successfully deleted
I20240501 14:01:11.723711 3527 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.729557 29096 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:11.736650 32224 raft_consensus.cc:1261] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.737444 31965 raft_consensus.cc:1261] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.738060 3498 consensus_queue.cc:1035] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.738507 2856 consensus_queue.cc:1035] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.744679 29096 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]). Expected version 1 got 0
I20240501 14:01:11.746064 29105 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]). Expected version 1 got 0
I20240501 14:01:11.748350 3569 mvcc.cc:205] Tried to move back new op lower bound from 7022887205828157440 to 7022887205172277248. Current Snapshot: MvccSnapshot[applied={T|T < 7022887205828157440}]
I20240501 14:01:11.748683 3574 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:11.749495 3574 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:11.760890 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db] alter complete (version 1)
I20240501 14:01:11.765612 3230 consensus_queue.cc:1035] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.775761 29105 catalog_manager.cc:5576] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743) does not have the latest schema for tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]). Expected version 1 got 0
I20240501 14:01:11.775772 3576 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:11.777341 3576 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:11.779608 3575 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:11.780246 3575 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:11.781709 29105 catalog_manager.cc:5576] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783) does not have the latest schema for tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db]). Expected version 1 got 0
I20240501 14:01:11.798301 3530 consensus_queue.cc:1035] T 9320d3f885be46e4bf165243eaef41aa P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.865706 29105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:11.876953 2856 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.877844 3574 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:11.878327 3574 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: MemRowSet was empty: no flush needed.
I20240501 14:01:11.881124 3575 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:11.881624 3575 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: MemRowSet was empty: no flush needed.
I20240501 14:01:11.892874 29105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table [id=d2f1498258e94cf38a624649e74909db] alter complete (version 2)
I20240501 14:01:11.893579 2856 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:11.915828 3576 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:11.916342 3576 tablet.cc:1620] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: MemRowSet was empty: no flush needed.
I20240501 14:01:12.028472 29097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:12.035408 29096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:12.037849 3574 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:12.048242 3575 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:12.049954 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec202bb41a44e05939f1ecdf33d8c38 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.050383 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec202bb41a44e05939f1ecdf33d8c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.059010 29096 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: rebalancer_dynamic_table_new_name [id=d2f1498258e94cf38a624649e74909db] alter complete (version 3)
I20240501 14:01:12.061653 29224 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec202bb41a44e05939f1ecdf33d8c38 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.061879 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec202bb41a44e05939f1ecdf33d8c38 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.062129 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec202bb41a44e05939f1ecdf33d8c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.062258 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec202bb41a44e05939f1ecdf33d8c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.069551 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 380d3038e25f4f7793ba0f8c9bd64678 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.070032 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d3038e25f4f7793ba0f8c9bd64678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.070813 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 380d3038e25f4f7793ba0f8c9bd64678 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.071296 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d3038e25f4f7793ba0f8c9bd64678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.072348 3607 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:12.074215 29226 tablet_service.cc:1467] Processing CreateTablet for tablet 380d3038e25f4f7793ba0f8c9bd64678 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.074625 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d3038e25f4f7793ba0f8c9bd64678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.079747 29225 tablet_service.cc:1467] Processing CreateTablet for tablet 3df490207f6248ce8a36687d71fedb85 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.080025 31944 tablet_service.cc:1467] Processing CreateTablet for tablet 3df490207f6248ce8a36687d71fedb85 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.080173 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df490207f6248ce8a36687d71fedb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.080452 31944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df490207f6248ce8a36687d71fedb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.087299 3608 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:12.088801 3608 tablet_bootstrap.cc:654] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.088867 3607 tablet_bootstrap.cc:654] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.089352 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 3df490207f6248ce8a36687d71fedb85 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:12.089782 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df490207f6248ce8a36687d71fedb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.093719 3610 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:12.097084 3612 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:12.102032 3610 tablet_bootstrap.cc:654] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.103708 3612 tablet_bootstrap.cc:654] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.105104 3607 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: No bootstrap required, opened a new log
I20240501 14:01:12.105341 3607 ts_tablet_manager.cc:1367] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 14:01:12.106168 3607 raft_consensus.cc:348] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.106524 3607 raft_consensus.cc:374] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.106673 3607 raft_consensus.cc:729] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.106988 3607 consensus_queue.cc:260] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.107869 3607 ts_tablet_manager.cc:1398] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:12.111871 3576 tablet.cc:1722] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:12.119272 3608 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:12.121187 3608 ts_tablet_manager.cc:1367] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.011s
I20240501 14:01:12.121853 3609 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:12.123755 3608 raft_consensus.cc:348] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.125933 3608 raft_consensus.cc:374] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.126138 3608 raft_consensus.cc:729] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.127261 3609 tablet_bootstrap.cc:654] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.128585 3612 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79: No bootstrap required, opened a new log
I20240501 14:01:12.129304 3612 ts_tablet_manager.cc:1367] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240501 14:01:12.130540 3612 raft_consensus.cc:348] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.132917 3608 consensus_queue.cc:260] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.135036 3612 raft_consensus.cc:374] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.135156 3612 raft_consensus.cc:729] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.135402 3612 consensus_queue.cc:260] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.136035 3612 ts_tablet_manager.cc:1398] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:01:12.137918 3608 ts_tablet_manager.cc:1398] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:01:12.157274 3608 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:12.160557 3608 tablet_bootstrap.cc:654] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.166357 3609 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:12.166606 3609 ts_tablet_manager.cc:1367] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.006s
I20240501 14:01:12.167083 3608 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:12.167577 3608 ts_tablet_manager.cc:1367] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:01:12.167465 3609 raft_consensus.cc:348] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.168006 3609 raft_consensus.cc:374] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.168115 3609 raft_consensus.cc:729] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.168407 3609 consensus_queue.cc:260] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.169474 29103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:12.169670 29103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59850:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:12.171388 3608 raft_consensus.cc:348] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.173362 3608 raft_consensus.cc:374] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.173588 3608 raft_consensus.cc:729] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.174103 3608 consensus_queue.cc:260] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.176559 3608 ts_tablet_manager.cc:1398] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:01:12.178696 29103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 589b051e6abe4fa38c771a41c6df7284: Sending DeleteTablet for 3 replicas of tablet a899520b90754b0e886db727ad89574a
I20240501 14:01:12.179383 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet a899520b90754b0e886db727ad89574a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:12 UTC) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:12.180088 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet a899520b90754b0e886db727ad89574a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:12.180768 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet a899520b90754b0e886db727ad89574a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:12 UTC) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:12.192713 3620 tablet_replica.cc:330] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:12.193173 3620 raft_consensus.cc:2229] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:12.193670 3620 raft_consensus.cc:2258] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.198668 3608 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:12.200642 3608 tablet_bootstrap.cc:654] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.201189 3139 raft_consensus.cc:482] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.201355 3139 raft_consensus.cc:504] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.201845 3621 tablet_replica.cc:330] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:12.202090 3139 leader_election.cc:288] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:12.202127 3621 raft_consensus.cc:2229] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.202459 3621 raft_consensus.cc:2258] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.202818 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:12.203243 3622 tablet_replica.cc:330] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:12.203514 3622 raft_consensus.cc:2229] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 14:01:12.203665 32161 leader_election.cc:341] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:12.203886 3622 raft_consensus.cc:2258] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.204238 3609 ts_tablet_manager.cc:1398] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240501 14:01:12.204623 3620 ts_tablet_manager.cc:1875] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:01:12.207038 32161 leader_election.cc:341] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Not found: Tablet not found: 380d3038e25f4f7793ba0f8c9bd64678
I20240501 14:01:12.207206 32161 leader_election.cc:302] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:12.207636 3519 raft_consensus.cc:2726] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:12.208611 3621 ts_tablet_manager.cc:1875] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:12.209925 3622 ts_tablet_manager.cc:1875] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:12.216706 3609 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:12.222222 3610 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7: No bootstrap required, opened a new log
I20240501 14:01:12.222909 3610 ts_tablet_manager.cc:1367] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.005s
I20240501 14:01:12.224063 3610 raft_consensus.cc:348] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.224450 3610 raft_consensus.cc:374] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.224592 3610 raft_consensus.cc:729] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.224985 3610 consensus_queue.cc:260] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.234804 3609 tablet_bootstrap.cc:654] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.236182 3623 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:12.237515 3610 ts_tablet_manager.cc:1398] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:12.238106 3623 tablet_bootstrap.cc:654] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.243139 3620 ts_tablet_manager.cc:1888] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:12.255085 3608 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba: No bootstrap required, opened a new log
I20240501 14:01:12.255331 3608 ts_tablet_manager.cc:1367] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20240501 14:01:12.256218 3608 raft_consensus.cc:348] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.256464 3608 raft_consensus.cc:374] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.256548 3608 raft_consensus.cc:729] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.256796 3608 consensus_queue.cc:260] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.257530 3608 ts_tablet_manager.cc:1398] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:12.260849 3621 ts_tablet_manager.cc:1888] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:12.262990 3622 ts_tablet_manager.cc:1888] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:12.263137 3622 log.cc:1196] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/a899520b90754b0e886db727ad89574a
I20240501 14:01:12.263648 3622 ts_tablet_manager.cc:1909] T a899520b90754b0e886db727ad89574a P 6a446ac802a7457d9f27076781f21d79: Deleting consensus metadata
I20240501 14:01:12.263774 3620 log.cc:1196] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/a899520b90754b0e886db727ad89574a
I20240501 14:01:12.264261 3620 ts_tablet_manager.cc:1909] T a899520b90754b0e886db727ad89574a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting consensus metadata
I20240501 14:01:12.268895 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table_new_name [id=d2f1498258e94cf38a624649e74909db]) successfully deleted
I20240501 14:01:12.280723 3621 log.cc:1196] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/a899520b90754b0e886db727ad89574a
I20240501 14:01:12.280872 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table_new_name [id=d2f1498258e94cf38a624649e74909db]) successfully deleted
I20240501 14:01:12.282121 3621 ts_tablet_manager.cc:1909] T a899520b90754b0e886db727ad89574a P 9903ed35bca144bcb74e9bf5102117e7: Deleting consensus metadata
I20240501 14:01:12.283444 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet a899520b90754b0e886db727ad89574a (table rebalancer_dynamic_table_new_name [id=d2f1498258e94cf38a624649e74909db]) successfully deleted
I20240501 14:01:12.287266 3521 raft_consensus.cc:482] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.287484 3521 raft_consensus.cc:504] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.288529 3521 leader_election.cc:288] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.288913 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec202bb41a44e05939f1ecdf33d8c38" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:12.289157 29377 raft_consensus.cc:2454] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:12.289319 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec202bb41a44e05939f1ecdf33d8c38" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:12.289544 32296 leader_election.cc:302] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:12.289585 29244 raft_consensus.cc:2454] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 0.
I20240501 14:01:12.289850 3521 raft_consensus.cc:2781] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:12.289952 3521 raft_consensus.cc:482] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:12.290059 3521 raft_consensus.cc:3037] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:12.290414 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 43677.5us
I20240501 14:01:12.292475 3521 raft_consensus.cc:504] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.293502 3521 leader_election.cc:288] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.293741 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec202bb41a44e05939f1ecdf33d8c38" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:12.293788 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec202bb41a44e05939f1ecdf33d8c38" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:12.293933 29377 raft_consensus.cc:3037] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.294018 29246 raft_consensus.cc:3037] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.296555 29377 raft_consensus.cc:2454] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:12.297129 32296 leader_election.cc:302] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:01:12.297389 3521 raft_consensus.cc:2781] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:12.297520 3521 raft_consensus.cc:686] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:12.297750 3521 consensus_queue.cc:237] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.303917 29246 raft_consensus.cc:2454] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:01:12.306143 29103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59850:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:12.312487 29097 catalog_manager.cc:5484] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:12.324797 3530 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.324956 3530 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.325407 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2580d876e543549ca1c5677ccbf32c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd3147f545ea4d31beb4a1a6a6b2e541]), partition=
I20240501 14:01:12.325562 3530 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.325901 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2580d876e543549ca1c5677ccbf32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.326701 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:12.326982 31964 raft_consensus.cc:2454] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:12.327421 29181 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:01:12.327688 3530 raft_consensus.cc:2781] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:12.327754 3530 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:12.327805 3530 raft_consensus.cc:3037] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.330427 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:12.330549 3530 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
W20240501 14:01:12.330979 29181 leader_election.cc:341] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:12.332872 3530 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.333269 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
W20240501 14:01:12.333736 29181 leader_election.cc:341] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:12.334059 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:12.334316 31965 raft_consensus.cc:3037] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.339301 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2580d876e543549ca1c5677ccbf32c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd3147f545ea4d31beb4a1a6a6b2e541]), partition=
I20240501 14:01:12.339323 3530 raft_consensus.cc:482] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.339460 3530 raft_consensus.cc:504] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.339744 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2580d876e543549ca1c5677ccbf32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.340409 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
W20240501 14:01:12.341150 29179 leader_election.cc:341] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:12.341472 3530 leader_election.cc:288] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:12.341929 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:01:12.342191 32224 raft_consensus.cc:2454] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 0.
I20240501 14:01:12.342597 29180 leader_election.cc:302] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters: ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:12.342942 3530 raft_consensus.cc:2781] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:12.343014 3530 raft_consensus.cc:482] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:12.343070 3530 raft_consensus.cc:3037] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.348510 3530 raft_consensus.cc:504] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.349579 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:12.349814 32224 raft_consensus.cc:3037] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.352344 32224 raft_consensus.cc:2454] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:12.352838 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2580d876e543549ca1c5677ccbf32c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd3147f545ea4d31beb4a1a6a6b2e541]), partition=
I20240501 14:01:12.353278 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2580d876e543549ca1c5677ccbf32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:12.353278 29180 leader_election.cc:302] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79; no voters:
I20240501 14:01:12.353569 3230 raft_consensus.cc:2781] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:12.354714 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
W20240501 14:01:12.355295 29179 leader_election.cc:341] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:12.355558 3230 raft_consensus.cc:686] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:12.355818 3230 consensus_queue.cc:237] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.361816 3530 leader_election.cc:288] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:12.365518 31965 raft_consensus.cc:2454] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:01:12.366175 29181 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters: b87d188552494b4590afc37a85b98c35
I20240501 14:01:12.373994 3635 raft_consensus.cc:2781] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:12.374133 3635 raft_consensus.cc:686] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:12.374323 29103 catalog_manager.cc:5484] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:12.374411 3635 consensus_queue.cc:237] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.388509 3623 tablet_bootstrap.cc:492] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:12.389083 3623 ts_tablet_manager.cc:1367] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.000s
I20240501 14:01:12.390398 3623 raft_consensus.cc:348] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.390671 3623 raft_consensus.cc:374] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.390740 3623 raft_consensus.cc:729] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.391000 3623 consensus_queue.cc:260] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:12.391646 3623 ts_tablet_manager.cc:1398] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:12.392055 3623 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:12.395591 3623 tablet_bootstrap.cc:654] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.401501 3638 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:12.404937 3623 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e: No bootstrap required, opened a new log
I20240501 14:01:12.405128 3623 ts_tablet_manager.cc:1367] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 14:01:12.407425 3609 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:12.405788 3623 raft_consensus.cc:348] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.407831 3623 raft_consensus.cc:374] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.407922 3623 raft_consensus.cc:729] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.407970 3609 ts_tablet_manager.cc:1367] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.191s user 0.004s sys 0.004s
I20240501 14:01:12.408188 3623 consensus_queue.cc:260] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.409107 3623 ts_tablet_manager.cc:1398] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:01:12.409296 3609 raft_consensus.cc:348] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.409795 3609 raft_consensus.cc:374] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.409936 3609 raft_consensus.cc:729] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.408275 29103 catalog_manager.cc:5484] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:12.439195 3609 consensus_queue.cc:260] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.441062 3609 ts_tablet_manager.cc:1398] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240501 14:01:12.442229 3638 tablet_bootstrap.cc:654] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.442562 3609 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:12.450213 3609 tablet_bootstrap.cc:654] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:12.464322 3638 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58: No bootstrap required, opened a new log
I20240501 14:01:12.464504 3638 ts_tablet_manager.cc:1367] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.000s
I20240501 14:01:12.465842 3638 raft_consensus.cc:348] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.466105 3638 raft_consensus.cc:374] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.466207 3638 raft_consensus.cc:729] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.477895 3609 tablet_bootstrap.cc:492] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35: No bootstrap required, opened a new log
I20240501 14:01:12.478108 3609 ts_tablet_manager.cc:1367] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20240501 14:01:12.478819 3609 raft_consensus.cc:348] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.479079 3638 consensus_queue.cc:260] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.479334 3609 raft_consensus.cc:374] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:12.479504 3609 raft_consensus.cc:729] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:12.486289 3638 ts_tablet_manager.cc:1398] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:01:12.486580 3609 consensus_queue.cc:260] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.487421 3609 ts_tablet_manager.cc:1398] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:01:12.690349 3390 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.690549 3390 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:12.691423 3390 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:12.693104 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:01:12.693456 31965 raft_consensus.cc:2379] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b87d188552494b4590afc37a85b98c35 in current term 1: Already voted for candidate 6518b7a04854458bb2fb23edc8b6f7ba in this term.
I20240501 14:01:12.695431 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:01:12.697010 29310 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87d188552494b4590afc37a85b98c35; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:12.698187 3390 raft_consensus.cc:3037] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.704155 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:12.704483 3557 raft_consensus.cc:482] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:12.704677 3557 raft_consensus.cc:504] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.705682 3557 leader_election.cc:288] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.706586 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2580d876e543549ca1c5677ccbf32c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:01:12.706885 32096 raft_consensus.cc:2454] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:12.708024 29572 leader_election.cc:302] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:12.708947 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2580d876e543549ca1c5677ccbf32c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:01:12.709256 29373 raft_consensus.cc:2454] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 0.
I20240501 14:01:12.710172 3557 raft_consensus.cc:2781] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:12.710306 3557 raft_consensus.cc:482] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:12.710413 3557 raft_consensus.cc:3037] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.716418 3527 consensus_queue.cc:1035] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:12.724630 3557 raft_consensus.cc:504] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.725359 3390 raft_consensus.cc:2726] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:12.727675 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2580d876e543549ca1c5677ccbf32c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:12.727898 32096 raft_consensus.cc:3037] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.731408 32096 raft_consensus.cc:2454] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:12.731953 29572 leader_election.cc:302] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:12.732275 2856 raft_consensus.cc:2781] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:12.736433 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2580d876e543549ca1c5677ccbf32c" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:12.736791 29377 raft_consensus.cc:3037] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.754421 29377 raft_consensus.cc:2454] T 0d2580d876e543549ca1c5677ccbf32c P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
I20240501 14:01:12.760509 3527 consensus_queue.cc:1035] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:12.761962 3557 leader_election.cc:288] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 1 election: Requested vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:12.779851 2856 raft_consensus.cc:686] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:12.780242 2856 consensus_queue.cc:237] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:12.787598 29635 raft_consensus.cc:3037] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:12.792238 3635 consensus_queue.cc:1035] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:12.805078 3230 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:12.806058 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 14:01:12.864825 32032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 53670.1us
I20240501 14:01:12.869463 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:01:12.885452 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 81761.4us
I20240501 14:01:12.893607 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:12.920274 3530 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:12.922098 29570 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 155313us
I20240501 14:01:12.923959 29101 catalog_manager.cc:5484] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:12.956410 3687 consensus_queue.cc:1035] T 380d3038e25f4f7793ba0f8c9bd64678 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:12.974439 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:13.012593 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:13.022157 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:13.076992 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 14:01:13.126542 32163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 55051.3us
I20240501 14:01:13.200634 3680 consensus_queue.cc:1035] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:13.214851 3678 consensus_queue.cc:1035] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:13.369338 32222 consensus_queue.cc:237] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:13.372385 29244 raft_consensus.cc:1261] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.372391 31963 raft_consensus.cc:1261] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.373076 3139 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.373394 3519 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:13.374758 32161 consensus_peers.cc:485] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c696bdff78504c0c93f39a7e07a40dcd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.376844 3139 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:13.377378 29244 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:13.381929 31965 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:13.383561 29101 catalog_manager.cc:5484] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:13.439200 29637 consensus_queue.cc:237] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:13.441550 29244 raft_consensus.cc:1261] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.441885 32222 raft_consensus.cc:1261] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.442458 3679 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.442781 3678 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:13.443033 29571 consensus_peers.cc:485] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 798c9c25344d4beb9a079fb776044a17. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.462284 3679 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:13.462867 32222 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:13.470011 29244 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:13.472700 29101 catalog_manager.cc:5484] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:13.748569 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:13.799055 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:13.805727 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:13.806140 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:13.824779 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:13.861037 3805 ts_tablet_manager.cc:902] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:13.866356 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:13.875190 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:13.930377 3805 tablet_copy_client.cc:308] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:01:13.931643 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet c696bdff78504c0c93f39a7e07a40dcd from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57857)
I20240501 14:01:13.931865 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet c696bdff78504c0c93f39a7e07a40dcd from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57857: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-c696bdff78504c0c93f39a7e07a40dcd
I20240501 14:01:13.934819 32248 tablet_copy_source_session.cc:205] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:13.936594 3805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c696bdff78504c0c93f39a7e07a40dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:13.937090 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:13.952337 3810 ts_tablet_manager.cc:902] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:13.944355 3805 tablet_copy_client.cc:783] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.009012 3805 tablet_copy_client.cc:647] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.012495 3805 tablet_copy_client.cc:523] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.031742 3805 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:14.043536 3810 tablet_copy_client.cc:308] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:14.044180 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 798c9c25344d4beb9a079fb776044a17 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:37257)
I20240501 14:01:14.044337 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 798c9c25344d4beb9a079fb776044a17 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:37257: session id = 6edd6a3430d44c38a19c2804e555059f-798c9c25344d4beb9a079fb776044a17
I20240501 14:01:14.047614 29657 tablet_copy_source_session.cc:205] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.048955 3810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798c9c25344d4beb9a079fb776044a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.059340 3810 tablet_copy_client.cc:783] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.059604 3810 tablet_copy_client.cc:647] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.065598 3805 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.066015 3805 tablet_bootstrap.cc:492] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:14.066265 3805 ts_tablet_manager.cc:1367] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20240501 14:01:14.066628 3810 tablet_copy_client.cc:523] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.067113 3805 raft_consensus.cc:348] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:14.067351 3805 raft_consensus.cc:729] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:14.067624 3805 consensus_queue.cc:260] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:14.068209 3805 ts_tablet_manager.cc:1398] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:14.071640 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-c696bdff78504c0c93f39a7e07a40dcd received from {username='slave'} at 127.22.33.68:57857
I20240501 14:01:14.071784 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-c696bdff78504c0c93f39a7e07a40dcd on tablet c696bdff78504c0c93f39a7e07a40dcd with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:14.072420 3810 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:14.095491 3810 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.095965 3810 tablet_bootstrap.cc:492] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:01:14.096230 3810 ts_tablet_manager.cc:1367] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.003s
I20240501 14:01:14.097213 3810 raft_consensus.cc:348] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:14.097442 3810 raft_consensus.cc:729] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:14.097749 3810 consensus_queue.cc:260] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:14.099707 3810 ts_tablet_manager.cc:1398] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:14.100378 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-798c9c25344d4beb9a079fb776044a17 received from {username='slave'} at 127.22.33.72:37257
I20240501 14:01:14.100490 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-798c9c25344d4beb9a079fb776044a17 on tablet 798c9c25344d4beb9a079fb776044a17 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:14.386936 29635 raft_consensus.cc:1203] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:14.446748 32362 raft_consensus.cc:1203] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:14.453871 29179 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 59041.2us
I20240501 14:01:14.467598 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:14.476387 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:14.481269 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:14.484937 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:14.526695 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:14.578428 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:14.596873 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:14.601756 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:14.748414 32222 consensus_queue.cc:237] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:14.750525 29637 raft_consensus.cc:1261] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:14.750914 31965 raft_consensus.cc:1261] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:14.751062 3701 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:14.751438 3139 consensus_queue.cc:1035] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:14.753697 3701 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:14.754274 29637 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:14.762182 29101 catalog_manager.cc:5484] T c696bdff78504c0c93f39a7e07a40dcd P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:14.770016 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet c696bdff78504c0c93f39a7e07a40dcd with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:14.770130 31965 raft_consensus.cc:2932] T c696bdff78504c0c93f39a7e07a40dcd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:14.773116 3905 tablet_replica.cc:330] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:14.773386 3905 raft_consensus.cc:2229] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:14.773612 3905 raft_consensus.cc:2258] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:14.774443 3905 ts_tablet_manager.cc:1875] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:14.781565 3905 ts_tablet_manager.cc:1888] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:14.785446 3905 log.cc:1196] T c696bdff78504c0c93f39a7e07a40dcd P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/c696bdff78504c0c93f39a7e07a40dcd
I20240501 14:01:14.786204 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet c696bdff78504c0c93f39a7e07a40dcd (table rebalancer_extra_table_12 [id=d3bd989be0cd4498819f6dbbdeb2b93a]) successfully deleted
I20240501 14:01:15.180581 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:15.189733 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:15.203203 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:15.227109 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:01:15.240882 29571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 85139.3us
W20240501 14:01:15.249329 29572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 86580us
I20240501 14:01:15.273236 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:15.327419 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:15.352959 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:15.370502 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:15.483943 32223 consensus_queue.cc:237] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:15.497273 31965 raft_consensus.cc:1261] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.497850 29246 raft_consensus.cc:1261] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.497869 3139 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.498497 3139 consensus_queue.cc:1035] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.515583 3139 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:15.520130 31964 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:15.522516 29244 raft_consensus.cc:2932] T 820be6be18af4781ace626f239379cff P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:15.523404 29098 catalog_manager.cc:5484] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:15.548166 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet 820be6be18af4781ace626f239379cff with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:15.569298 3997 tablet_replica.cc:330] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:15.569525 3997 raft_consensus.cc:2229] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:15.569818 3997 raft_consensus.cc:2258] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:15.573035 3997 ts_tablet_manager.cc:1875] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:15.579133 3997 ts_tablet_manager.cc:1888] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:15.579468 3997 log.cc:1196] T 820be6be18af4781ace626f239379cff P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/820be6be18af4781ace626f239379cff
I20240501 14:01:15.581137 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 820be6be18af4781ace626f239379cff (table rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]) successfully deleted
I20240501 14:01:15.836191 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:15.843766 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:15.901182 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:15.970851 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:15.972029 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:15.983812 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:16.037734 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:16.075783 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:16.201076 32097 consensus_queue.cc:237] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:16.203254 32223 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:16.203403 32360 raft_consensus.cc:1261] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:16.203828 3691 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.204133 3689 consensus_queue.cc:1035] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.207361 3691 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.208030 32360 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.208474 32223 raft_consensus.cc:2932] T 414c148db46543b786bfd8b0faecfe96 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.214406 29098 catalog_manager.cc:5484] T 414c148db46543b786bfd8b0faecfe96 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.225857 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 414c148db46543b786bfd8b0faecfe96 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:16.226694 4091 tablet_replica.cc:330] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:16.226998 4091 raft_consensus.cc:2229] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:16.227293 4091 raft_consensus.cc:2258] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:16.229972 4091 ts_tablet_manager.cc:1875] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:16.236799 4091 ts_tablet_manager.cc:1888] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 14:01:16.237023 4091 log.cc:1196] T 414c148db46543b786bfd8b0faecfe96 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/414c148db46543b786bfd8b0faecfe96
I20240501 14:01:16.238222 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 414c148db46543b786bfd8b0faecfe96 (table rebalancer_extra_table_0 [id=e0d0b90f25384d879d18bbfc7e4b20dd]) successfully deleted
I20240501 14:01:16.397025 32362 consensus_queue.cc:237] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:16.400099 29505 raft_consensus.cc:1261] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:16.400525 32296 consensus_peers.cc:485] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e83b7b4decde4e459e49de9eb05b8c0f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.401038 29244 raft_consensus.cc:1261] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.401728 3633 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.402226 3633 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.404470 3633 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.405059 29244 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.411739 29505 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.414939 29102 catalog_manager.cc:5484] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.515461 31963 consensus_queue.cc:237] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
W20240501 14:01:16.519155 31899 consensus_peers.cc:485] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59c54188ad9e40dd925b50a250dcea04. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.519636 29637 raft_consensus.cc:1261] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.519815 32362 raft_consensus.cc:1261] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.520573 3314 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.521090 3314 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.530596 3314 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.533232 29637 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.543486 29098 catalog_manager.cc:5484] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.550164 32362 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.582350 29244 consensus_queue.cc:237] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:16.584501 29637 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.584646 29505 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:16.585445 29181 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba20c01d741a414196972ced1601baf5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.586082 3953 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.586771 3953 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.595408 3953 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:16.595872 29637 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:16.602100 29098 catalog_manager.cc:5484] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:16.602639 29505 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:16.640878 29637 consensus_queue.cc:237] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:16.643211 29505 raft_consensus.cc:1261] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.643352 29377 raft_consensus.cc:1261] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.643954 3964 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:16.644631 29571 consensus_peers.cc:485] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 241a2fdfe954476ba9bd2cfeb4b931ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.644665 4238 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.647585 3964 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.648198 29377 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.652582 29505 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.654989 29098 catalog_manager.cc:5484] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.700302 32097 consensus_queue.cc:237] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:16.702749 32222 raft_consensus.cc:1261] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.703464 4087 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:16.704067 32032 consensus_peers.cc:485] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c84a393bd047d985927417f82bec7d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.704427 31963 raft_consensus.cc:1261] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.705160 4087 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.707751 3691 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.708319 31963 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.713420 32222 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.714223 29100 catalog_manager.cc:5484] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:16.758271 29505 consensus_queue.cc:237] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:16.760923 29377 raft_consensus.cc:1261] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.761066 29637 raft_consensus.cc:1261] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.761679 4275 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:16.762037 29441 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8427dee24ad04247b23b24570c73c92a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.762035 3630 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.764734 4275 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.765337 29377 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.769474 29637 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.770627 29102 catalog_manager.cc:5484] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:16.843297 29505 consensus_queue.cc:237] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:16.846041 32096 raft_consensus.cc:1261] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.846246 29377 raft_consensus.cc:1261] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.846539 3630 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:16.847409 29440 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e190940ae8b44ca2bf1ee496a9d624ba. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.848279 3776 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.851910 3776 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:16.852422 29377 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:16.859127 29100 catalog_manager.cc:5484] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:16.864063 32096 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:16.913607 29244 consensus_queue.cc:237] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:16.916255 29377 raft_consensus.cc:1261] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.917011 29505 raft_consensus.cc:1261] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:16.916927 29180 consensus_peers.cc:485] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca0a888dc37d44c69dea73f9ffd556f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.918146 3953 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:01:16.919629 4217 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.929212 4311 ts_tablet_manager.cc:902] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:16.931012 4217 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.934530 29505 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.943091 29377 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.958312 29102 catalog_manager.cc:5484] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:16.978106 4311 tablet_copy_client.cc:308] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:01:16.979223 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet 59c54188ad9e40dd925b50a250dcea04 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:48147)
I20240501 14:01:16.979387 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet 59c54188ad9e40dd925b50a250dcea04 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:48147: session id = 6a446ac802a7457d9f27076781f21d79-59c54188ad9e40dd925b50a250dcea04
I20240501 14:01:16.982404 31985 tablet_copy_source_session.cc:205] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:16.996389 4311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c54188ad9e40dd925b50a250dcea04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.003788 4321 ts_tablet_manager.cc:902] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:17.008216 4321 tablet_copy_client.cc:308] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:17.012914 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet ba20c01d741a414196972ced1601baf5 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36699)
I20240501 14:01:17.013093 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet ba20c01d741a414196972ced1601baf5 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36699: session id = 0a697f3a1ee544daa7073814ab845d58-ba20c01d741a414196972ced1601baf5
I20240501 14:01:17.016019 29266 tablet_copy_source_session.cc:205] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.019315 4321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba20c01d741a414196972ced1601baf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.020133 4311 tablet_copy_client.cc:783] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.020373 4316 ts_tablet_manager.cc:902] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:17.020679 4311 tablet_copy_client.cc:647] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.022904 4311 tablet_copy_client.cc:523] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.026481 4311 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:17.027123 4321 tablet_copy_client.cc:783] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.027374 4321 tablet_copy_client.cc:647] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.031211 4321 tablet_copy_client.cc:523] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.040540 4321 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:17.041818 4311 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.042202 4311 tablet_bootstrap.cc:492] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:01:17.042457 4311 ts_tablet_manager.cc:1367] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20240501 14:01:17.046329 4311 raft_consensus.cc:348] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:17.046520 4311 raft_consensus.cc:729] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.047992 4316 tablet_copy_client.cc:308] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:01:17.049027 4311 consensus_queue.cc:260] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:17.049299 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet e83b7b4decde4e459e49de9eb05b8c0f from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:59749)
I20240501 14:01:17.049585 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet e83b7b4decde4e459e49de9eb05b8c0f from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:59749: session id = b87d188552494b4590afc37a85b98c35-e83b7b4decde4e459e49de9eb05b8c0f
I20240501 14:01:17.050055 4311 ts_tablet_manager.cc:1398] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:01:17.050707 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-59c54188ad9e40dd925b50a250dcea04 received from {username='slave'} at 127.22.33.71:48147
I20240501 14:01:17.050854 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-59c54188ad9e40dd925b50a250dcea04 on tablet 59c54188ad9e40dd925b50a250dcea04 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:17.058408 32383 tablet_copy_source_session.cc:205] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.060566 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83b7b4decde4e459e49de9eb05b8c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.067322 4316 tablet_copy_client.cc:783] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.067554 4316 tablet_copy_client.cc:647] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.071087 4316 tablet_copy_client.cc:523] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.074577 4316 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:17.108779 4316 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.109207 4316 tablet_bootstrap.cc:492] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:17.109453 4316 ts_tablet_manager.cc:1367] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.013s
I20240501 14:01:17.110177 4316 raft_consensus.cc:348] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.110400 4316 raft_consensus.cc:729] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.110616 4316 consensus_queue.cc:260] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.111176 4316 ts_tablet_manager.cc:1398] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:17.118582 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-e83b7b4decde4e459e49de9eb05b8c0f received from {username='slave'} at 127.22.33.66:59749
I20240501 14:01:17.118824 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session b87d188552494b4590afc37a85b98c35-e83b7b4decde4e459e49de9eb05b8c0f on tablet e83b7b4decde4e459e49de9eb05b8c0f with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:17.124250 4321 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.124646 4321 tablet_bootstrap.cc:492] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:17.124877 4321 ts_tablet_manager.cc:1367] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.084s user 0.056s sys 0.015s
I20240501 14:01:17.125797 4321 raft_consensus.cc:348] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:17.126063 4321 raft_consensus.cc:729] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.126354 4321 consensus_queue.cc:260] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:17.127049 4321 ts_tablet_manager.cc:1398] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20240501 14:01:17.127923 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-ba20c01d741a414196972ced1601baf5 received from {username='slave'} at 127.22.33.70:36699
I20240501 14:01:17.128043 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-ba20c01d741a414196972ced1601baf5 on tablet ba20c01d741a414196972ced1601baf5 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:17.128526 4343 ts_tablet_manager.cc:902] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:17.136049 4311 ts_tablet_manager.cc:902] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:17.173442 4343 tablet_copy_client.cc:308] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:17.174278 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 241a2fdfe954476ba9bd2cfeb4b931ac from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:37257)
I20240501 14:01:17.174446 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 241a2fdfe954476ba9bd2cfeb4b931ac from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:37257: session id = 6edd6a3430d44c38a19c2804e555059f-241a2fdfe954476ba9bd2cfeb4b931ac
I20240501 14:01:17.177188 29657 tablet_copy_source_session.cc:205] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.178798 4343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241a2fdfe954476ba9bd2cfeb4b931ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.187263 4343 tablet_copy_client.cc:783] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.187585 4343 tablet_copy_client.cc:647] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.191401 4343 tablet_copy_client.cc:523] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.196615 4343 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:17.210773 4311 tablet_copy_client.cc:308] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:17.211771 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 8427dee24ad04247b23b24570c73c92a from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:41457)
I20240501 14:01:17.211920 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 8427dee24ad04247b23b24570c73c92a from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:41457: session id = 6a446ac802a7457d9f27076781f21d79-8427dee24ad04247b23b24570c73c92a
I20240501 14:01:17.214200 29527 tablet_copy_source_session.cc:205] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.215510 4311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8427dee24ad04247b23b24570c73c92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.223248 4311 tablet_copy_client.cc:783] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.223527 4311 tablet_copy_client.cc:647] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.224215 32360 consensus_queue.cc:237] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.226514 4311 tablet_copy_client.cc:523] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.231266 4311 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:17.240726 4316 ts_tablet_manager.cc:902] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:17.261605 4343 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.262056 4343 tablet_bootstrap.cc:492] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:01:17.262313 4343 ts_tablet_manager.cc:1367] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.066s user 0.040s sys 0.018s
I20240501 14:01:17.262987 4343 raft_consensus.cc:348] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:17.263195 4343 raft_consensus.cc:729] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.263442 4343 consensus_queue.cc:260] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:17.264010 4343 ts_tablet_manager.cc:1398] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:17.264663 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-241a2fdfe954476ba9bd2cfeb4b931ac received from {username='slave'} at 127.22.33.72:37257
I20240501 14:01:17.264781 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-241a2fdfe954476ba9bd2cfeb4b931ac on tablet 241a2fdfe954476ba9bd2cfeb4b931ac with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:17.279259 4316 tablet_copy_client.cc:308] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:17.280161 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 62c84a393bd047d985927417f82bec7d from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:42885)
I20240501 14:01:17.280346 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 62c84a393bd047d985927417f82bec7d from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:42885: session id = b87d188552494b4590afc37a85b98c35-62c84a393bd047d985927417f82bec7d
I20240501 14:01:17.282830 29637 raft_consensus.cc:1261] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.282986 32117 tablet_copy_source_session.cc:205] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.283428 4057 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.284042 31963 raft_consensus.cc:1261] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.284778 4057 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:17.285558 32296 consensus_peers.cc:485] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7c0ef199394de490b790a26532df3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.287344 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c84a393bd047d985927417f82bec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.290261 4180 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.293723 4316 tablet_copy_client.cc:783] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.293939 4316 tablet_copy_client.cc:647] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.295822 31963 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.301326 4316 tablet_copy_client.cc:523] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.305141 4316 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:17.305114 29636 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.305279 29102 catalog_manager.cc:5484] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.310887 4316 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.311239 4316 tablet_bootstrap.cc:492] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:17.311445 4316 ts_tablet_manager.cc:1367] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:01:17.312188 4316 raft_consensus.cc:348] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.312398 4316 raft_consensus.cc:729] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.312609 4316 consensus_queue.cc:260] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.313232 4316 ts_tablet_manager.cc:1398] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:17.314316 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-62c84a393bd047d985927417f82bec7d received from {username='slave'} at 127.22.33.66:42885
I20240501 14:01:17.314447 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session b87d188552494b4590afc37a85b98c35-62c84a393bd047d985927417f82bec7d on tablet 62c84a393bd047d985927417f82bec7d with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:17.325417 4311 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.325939 4311 tablet_bootstrap.cc:492] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:01:17.326336 4311 ts_tablet_manager.cc:1367] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.095s user 0.063s sys 0.021s
I20240501 14:01:17.327402 4311 raft_consensus.cc:348] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:17.327657 4311 raft_consensus.cc:729] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.329916 4311 consensus_queue.cc:260] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:17.337787 4311 ts_tablet_manager.cc:1398] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20240501 14:01:17.339035 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-8427dee24ad04247b23b24570c73c92a received from {username='slave'} at 127.22.33.71:41457
I20240501 14:01:17.339152 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-8427dee24ad04247b23b24570c73c92a on tablet 8427dee24ad04247b23b24570c73c92a with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:17.371083 29376 raft_consensus.cc:1203] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.405711 29376 consensus_queue.cc:237] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
W20240501 14:01:17.408319 29310 consensus_peers.cc:485] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb3c16421adb477e821495e785849cc4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.408318 32360 raft_consensus.cc:1261] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.409039 3879 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.412796 31963 raft_consensus.cc:1261] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.421211 32097 raft_consensus.cc:1203] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:17.422533 4388 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.430922 32223 raft_consensus.cc:1203] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.430951 3879 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.433909 4343 ts_tablet_manager.cc:902] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:17.435953 32360 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.436862 31963 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.438992 29101 catalog_manager.cc:5484] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.457669 4343 tablet_copy_client.cc:308] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:17.458401 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet ca0a888dc37d44c69dea73f9ffd556f6 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:50987)
I20240501 14:01:17.458557 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet ca0a888dc37d44c69dea73f9ffd556f6 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:50987: session id = 6edd6a3430d44c38a19c2804e555059f-ca0a888dc37d44c69dea73f9ffd556f6
I20240501 14:01:17.461586 29266 tablet_copy_source_session.cc:205] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.464432 4343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0a888dc37d44c69dea73f9ffd556f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.481564 4343 tablet_copy_client.cc:783] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.481846 4343 tablet_copy_client.cc:647] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.488679 4343 tablet_copy_client.cc:523] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.492200 4343 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:17.501329 32360 raft_consensus.cc:1203] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:17.509550 4402 ts_tablet_manager.cc:902] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:17.546829 4402 tablet_copy_client.cc:308] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:17.548405 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet e190940ae8b44ca2bf1ee496a9d624ba from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:44365)
I20240501 14:01:17.548540 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet e190940ae8b44ca2bf1ee496a9d624ba from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:44365: session id = 6518b7a04854458bb2fb23edc8b6f7ba-e190940ae8b44ca2bf1ee496a9d624ba
I20240501 14:01:17.555882 29527 tablet_copy_source_session.cc:205] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.560603 32222 raft_consensus.cc:1203] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:17.561635 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190940ae8b44ca2bf1ee496a9d624ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.576615 4402 tablet_copy_client.cc:783] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.577155 4402 tablet_copy_client.cc:647] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.580063 4402 tablet_copy_client.cc:523] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.582940 4343 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.583611 4343 tablet_bootstrap.cc:492] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:01:17.583933 4343 ts_tablet_manager.cc:1367] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.092s user 0.055s sys 0.007s
I20240501 14:01:17.586019 4343 raft_consensus.cc:348] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:17.586313 4343 raft_consensus.cc:729] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.586810 4402 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:17.586699 4343 consensus_queue.cc:260] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:17.588835 4343 ts_tablet_manager.cc:1398] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:17.590054 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-ca0a888dc37d44c69dea73f9ffd556f6 received from {username='slave'} at 127.22.33.72:50987
I20240501 14:01:17.590194 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-ca0a888dc37d44c69dea73f9ffd556f6 on tablet ca0a888dc37d44c69dea73f9ffd556f6 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:17.604869 4402 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.605319 4402 tablet_bootstrap.cc:492] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:17.605564 4402 ts_tablet_manager.cc:1367] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20240501 14:01:17.606330 4402 raft_consensus.cc:348] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:17.606520 4402 raft_consensus.cc:729] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.606799 4402 consensus_queue.cc:260] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:17.607519 4402 ts_tablet_manager.cc:1398] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:17.608253 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-e190940ae8b44ca2bf1ee496a9d624ba received from {username='slave'} at 127.22.33.65:44365
I20240501 14:01:17.608368 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-e190940ae8b44ca2bf1ee496a9d624ba on tablet e190940ae8b44ca2bf1ee496a9d624ba with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:17.615020 32222 consensus_queue.cc:237] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:17.616951 29376 raft_consensus.cc:1203] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.622287 29636 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.622488 32360 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.623029 4370 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.623359 3139 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:17.623684 32162 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fbb60619c1a4d8e8cd6266596a0614d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.627058 4370 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:17.627647 32360 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:17.632742 29636 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:17.636340 29100 catalog_manager.cc:5484] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:17.698952 29636 consensus_queue.cc:237] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.702195 32222 raft_consensus.cc:1261] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.702689 32097 raft_consensus.cc:1261] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.703644 4238 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:17.704612 29572 consensus_peers.cc:485] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34d3876037334884a4a36bc192956cf4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.704872 4238 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.718557 4238 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.719548 32097 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.724280 32222 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.730120 29101 catalog_manager.cc:5484] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:17.783427 32096 consensus_queue.cc:237] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
W20240501 14:01:17.792757 32030 consensus_peers.cc:485] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ff2fc8d230a45daa4e0c8ba4fec422e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.793114 32360 raft_consensus.cc:1261] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.793571 29376 raft_consensus.cc:1261] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.793784 4087 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.794337 4087 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.798506 4087 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:17.799145 32360 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:17.807070 29098 catalog_manager.cc:5484] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:17.809572 4316 ts_tablet_manager.cc:902] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:17.814653 29376 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:17.834339 4316 tablet_copy_client.cc:308] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:01:17.835242 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet 9c7c0ef199394de490b790a26532df3b from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:59749)
I20240501 14:01:17.835443 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet 9c7c0ef199394de490b790a26532df3b from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:59749: session id = b87d188552494b4590afc37a85b98c35-9c7c0ef199394de490b790a26532df3b
I20240501 14:01:17.838183 32383 tablet_copy_source_session.cc:205] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.842375 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7c0ef199394de490b790a26532df3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.848841 4316 tablet_copy_client.cc:783] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.849112 4316 tablet_copy_client.cc:647] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.850939 4316 tablet_copy_client.cc:523] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.855293 4316 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:17.861860 4316 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.862622 4316 tablet_bootstrap.cc:492] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:17.862859 4316 ts_tablet_manager.cc:1367] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.004s
I20240501 14:01:17.863572 4316 raft_consensus.cc:348] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.863776 4316 raft_consensus.cc:729] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:17.863967 4316 consensus_queue.cc:260] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:17.864463 4316 ts_tablet_manager.cc:1398] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:17.865087 32096 consensus_queue.cc:237] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:17.866325 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-9c7c0ef199394de490b790a26532df3b received from {username='slave'} at 127.22.33.66:59749
I20240501 14:01:17.866443 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session b87d188552494b4590afc37a85b98c35-9c7c0ef199394de490b790a26532df3b on tablet 9c7c0ef199394de490b790a26532df3b with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:17.868310 31964 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:17.868643 32030 consensus_peers.cc:485] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb982ed7bcf74eb981c5199d87f9b4c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.869216 4462 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.869424 32360 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.870615 3691 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.890559 4462 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.897192 31964 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.904284 32360 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.904862 29101 catalog_manager.cc:5484] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:17.943938 4402 ts_tablet_manager.cc:902] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:17.991510 32360 raft_consensus.cc:1203] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:17.993938 4402 tablet_copy_client.cc:308] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:17.994663 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet eb3c16421adb477e821495e785849cc4 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:35753)
I20240501 14:01:17.994808 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet eb3c16421adb477e821495e785849cc4 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:35753: session id = 6518b7a04854458bb2fb23edc8b6f7ba-eb3c16421adb477e821495e785849cc4
I20240501 14:01:17.997259 29397 tablet_copy_source_session.cc:205] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.998544 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3c16421adb477e821495e785849cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.005823 4402 tablet_copy_client.cc:783] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.006193 4402 tablet_copy_client.cc:647] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.007768 4402 tablet_copy_client.cc:523] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.010669 4402 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:18.017431 4402 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.017825 4402 tablet_bootstrap.cc:492] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:18.018118 4402 ts_tablet_manager.cc:1367] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240501 14:01:18.018755 4402 raft_consensus.cc:348] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:18.018965 4402 raft_consensus.cc:729] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.019171 4402 consensus_queue.cc:260] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:18.021497 4402 ts_tablet_manager.cc:1398] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:18.025806 29244 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:18.030160 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-eb3c16421adb477e821495e785849cc4 received from {username='slave'} at 127.22.33.65:35753
I20240501 14:01:18.030280 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-eb3c16421adb477e821495e785849cc4 on tablet eb3c16421adb477e821495e785849cc4 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:18.030970 29373 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:18.031324 29180 consensus_peers.cc:485] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500b9c9bdfa649bd883d6ef2d6e541fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.031983 4217 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.032924 29507 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.034263 4217 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.045033 4217 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:18.045557 29373 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:18.051543 29100 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:18.057809 29505 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:18.067960 29244 raft_consensus.cc:1203] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.102488 31964 consensus_queue.cc:237] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:18.105057 32096 raft_consensus.cc:1261] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:18.105109 32222 raft_consensus.cc:1261] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:18.105741 4390 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:18.106038 31900 consensus_peers.cc:485] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfbf43ae81ae45ba8d5abd345d18ac8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.106354 4390 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:18.108893 4200 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:18.109506 32096 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:18.112937 32222 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:18.115377 29098 catalog_manager.cc:5484] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:18.144088 4316 ts_tablet_manager.cc:902] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:18.171136 4316 tablet_copy_client.cc:308] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:18.172276 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 34d3876037334884a4a36bc192956cf4 from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:59997)
I20240501 14:01:18.172425 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 34d3876037334884a4a36bc192956cf4 from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:59997: session id = b87d188552494b4590afc37a85b98c35-34d3876037334884a4a36bc192956cf4
I20240501 14:01:18.176894 29657 tablet_copy_source_session.cc:205] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.180233 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3876037334884a4a36bc192956cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.186794 4316 tablet_copy_client.cc:783] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.187062 4316 tablet_copy_client.cc:647] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.189013 4316 tablet_copy_client.cc:523] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.192147 4316 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:18.198062 4316 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.198438 4316 tablet_bootstrap.cc:492] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:18.198663 4316 ts_tablet_manager.cc:1367] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 14:01:18.199226 4316 raft_consensus.cc:348] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:18.199435 4316 raft_consensus.cc:729] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.199677 4316 consensus_queue.cc:260] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:18.200258 4316 ts_tablet_manager.cc:1398] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:18.201126 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-34d3876037334884a4a36bc192956cf4 received from {username='slave'} at 127.22.33.66:59997
I20240501 14:01:18.201252 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session b87d188552494b4590afc37a85b98c35-34d3876037334884a4a36bc192956cf4 on tablet 34d3876037334884a4a36bc192956cf4 with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:18.206699 4546 ts_tablet_manager.cc:902] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:18.229573 4546 tablet_copy_client.cc:308] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:18.230283 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57709)
I20240501 14:01:18.230468 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 0ff2fc8d230a45daa4e0c8ba4fec422e from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57709: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-0ff2fc8d230a45daa4e0c8ba4fec422e
I20240501 14:01:18.232790 32117 tablet_copy_source_session.cc:205] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.234226 4546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.234421 4555 ts_tablet_manager.cc:902] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:18.243203 4546 tablet_copy_client.cc:783] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.243523 4546 tablet_copy_client.cc:647] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.245635 4546 tablet_copy_client.cc:523] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.249830 4555 tablet_copy_client.cc:308] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:01:18.250743 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet 6fbb60619c1a4d8e8cd6266596a0614d from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:38119)
I20240501 14:01:18.250941 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet 6fbb60619c1a4d8e8cd6266596a0614d from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:38119: session id = 522f4ff5e9084cbf8462196533c7d7aa-6fbb60619c1a4d8e8cd6266596a0614d
I20240501 14:01:18.251374 4546 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:18.253971 32248 tablet_copy_source_session.cc:205] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.255659 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbb60619c1a4d8e8cd6266596a0614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.263916 4555 tablet_copy_client.cc:783] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.264231 4555 tablet_copy_client.cc:647] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.264420 4546 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.264813 4546 tablet_bootstrap.cc:492] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:18.265084 4546 ts_tablet_manager.cc:1367] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.010s
I20240501 14:01:18.265630 4546 raft_consensus.cc:348] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:18.265810 4546 raft_consensus.cc:729] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.265812 4555 tablet_copy_client.cc:523] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.266019 4546 consensus_queue.cc:260] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:18.266535 4546 ts_tablet_manager.cc:1398] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:18.268607 4555 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:18.268743 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-0ff2fc8d230a45daa4e0c8ba4fec422e received from {username='slave'} at 127.22.33.68:57709
I20240501 14:01:18.268894 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-0ff2fc8d230a45daa4e0c8ba4fec422e on tablet 0ff2fc8d230a45daa4e0c8ba4fec422e with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:18.283650 4555 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.284040 4555 tablet_bootstrap.cc:492] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:01:18.284273 4555 ts_tablet_manager.cc:1367] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240501 14:01:18.284973 4555 raft_consensus.cc:348] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:18.285188 4555 raft_consensus.cc:729] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.285430 4555 consensus_queue.cc:260] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:18.286055 4555 ts_tablet_manager.cc:1398] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:18.286768 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-6fbb60619c1a4d8e8cd6266596a0614d received from {username='slave'} at 127.22.33.67:38119
I20240501 14:01:18.286929 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-6fbb60619c1a4d8e8cd6266596a0614d on tablet 6fbb60619c1a4d8e8cd6266596a0614d with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:01:18.334964 29377 raft_consensus.cc:1203] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.411588 4402 ts_tablet_manager.cc:902] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:18.448437 4402 tablet_copy_client.cc:308] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:18.453613 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet eb982ed7bcf74eb981c5199d87f9b4c4 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:54889)
I20240501 14:01:18.453840 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet eb982ed7bcf74eb981c5199d87f9b4c4 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:54889: session id = 6518b7a04854458bb2fb23edc8b6f7ba-eb982ed7bcf74eb981c5199d87f9b4c4
I20240501 14:01:18.456645 32117 tablet_copy_source_session.cc:205] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.462913 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb982ed7bcf74eb981c5199d87f9b4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.469959 4402 tablet_copy_client.cc:783] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.470237 4402 tablet_copy_client.cc:647] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.472434 4402 tablet_copy_client.cc:523] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.478261 29246 raft_consensus.cc:1203] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.484112 4402 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:18.522629 4402 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.525453 4402 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:18.526130 4402 ts_tablet_manager.cc:1367] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.008s
I20240501 14:01:18.528869 4402 raft_consensus.cc:348] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:18.530781 4402 raft_consensus.cc:729] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.531378 4402 consensus_queue.cc:260] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:18.538468 4402 ts_tablet_manager.cc:1398] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:18.539547 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-eb982ed7bcf74eb981c5199d87f9b4c4 received from {username='slave'} at 127.22.33.65:54889
I20240501 14:01:18.539707 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-eb982ed7bcf74eb981c5199d87f9b4c4 on tablet eb982ed7bcf74eb981c5199d87f9b4c4 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:18.571022 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:18.573714 4617 ts_tablet_manager.cc:902] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:18.574268 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:18.575542 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:18.608140 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:18.629110 4316 ts_tablet_manager.cc:902] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:18.630013 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:18.648340 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:18.656574 29507 raft_consensus.cc:1203] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.684718 29637 raft_consensus.cc:1203] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.709841 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:18.711259 4316 tablet_copy_client.cc:308] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:01:18.712121 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet bfbf43ae81ae45ba8d5abd345d18ac8a from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:57827)
I20240501 14:01:18.712333 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet bfbf43ae81ae45ba8d5abd345d18ac8a from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:57827: session id = b87d188552494b4590afc37a85b98c35-bfbf43ae81ae45ba8d5abd345d18ac8a
I20240501 14:01:18.716409 31985 tablet_copy_source_session.cc:205] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.721489 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbf43ae81ae45ba8d5abd345d18ac8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.731237 29377 raft_consensus.cc:1203] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.754453 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:18.784991 4617 tablet_copy_client.cc:308] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:18.785642 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 500b9c9bdfa649bd883d6ef2d6e541fc from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:34317)
I20240501 14:01:18.785799 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 500b9c9bdfa649bd883d6ef2d6e541fc from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:34317: session id = 6a446ac802a7457d9f27076781f21d79-500b9c9bdfa649bd883d6ef2d6e541fc
I20240501 14:01:18.788524 29266 tablet_copy_source_session.cc:205] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.790091 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500b9c9bdfa649bd883d6ef2d6e541fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.799978 4617 tablet_copy_client.cc:783] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.800345 4617 tablet_copy_client.cc:647] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.802232 4617 tablet_copy_client.cc:523] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.808133 4617 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:18.877215 4316 tablet_copy_client.cc:783] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.877480 4316 tablet_copy_client.cc:647] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.884159 4316 tablet_copy_client.cc:523] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.891844 4316 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:18.926045 4617 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.926642 4617 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:01:18.927026 4617 ts_tablet_manager.cc:1367] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.119s user 0.037s sys 0.020s
I20240501 14:01:18.927932 4617 raft_consensus.cc:348] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:18.928251 4617 raft_consensus.cc:729] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.928562 4617 consensus_queue.cc:260] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:18.930016 4617 ts_tablet_manager.cc:1398] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:18.934830 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-500b9c9bdfa649bd883d6ef2d6e541fc received from {username='slave'} at 127.22.33.71:34317
I20240501 14:01:18.934962 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-500b9c9bdfa649bd883d6ef2d6e541fc on tablet 500b9c9bdfa649bd883d6ef2d6e541fc with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:18.941843 4316 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.942273 4316 tablet_bootstrap.cc:492] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:18.942520 4316 ts_tablet_manager.cc:1367] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.007s
I20240501 14:01:18.943392 4316 raft_consensus.cc:348] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:18.943617 4316 raft_consensus.cc:729] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:18.943914 4316 consensus_queue.cc:260] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:18.955026 4316 ts_tablet_manager.cc:1398] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 14:01:18.956596 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-bfbf43ae81ae45ba8d5abd345d18ac8a received from {username='slave'} at 127.22.33.66:57827
I20240501 14:01:18.956730 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session b87d188552494b4590afc37a85b98c35-bfbf43ae81ae45ba8d5abd345d18ac8a on tablet bfbf43ae81ae45ba8d5abd345d18ac8a with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:19.002650 29244 raft_consensus.cc:1203] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:19.051323 32222 raft_consensus.cc:1203] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:19.121747 29377 raft_consensus.cc:1203] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:19.404489 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:19.417587 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:19.493148 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:19.525362 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:19.539333 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:19.549266 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:19.622162 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:19.649888 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:19.771287 32097 consensus_queue.cc:237] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:19.773438 29244 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:19.773474 32362 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:19.774055 4462 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:19.774461 4087 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:19.777745 4462 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:19.778468 32362 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:19.782393 29244 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:19.789223 29103 catalog_manager.cc:5484] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:19.823661 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet eb982ed7bcf74eb981c5199d87f9b4c4 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:19.837554 4735 tablet_replica.cc:330] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:19.837961 4735 raft_consensus.cc:2229] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:19.838335 4735 raft_consensus.cc:2258] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:19.839591 4735 ts_tablet_manager.cc:1875] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:19.848528 4735 ts_tablet_manager.cc:1888] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:19.848788 4735 log.cc:1196] T eb982ed7bcf74eb981c5199d87f9b4c4 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb982ed7bcf74eb981c5199d87f9b4c4
I20240501 14:01:19.850986 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet eb982ed7bcf74eb981c5199d87f9b4c4 (table rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]) successfully deleted
I20240501 14:01:20.112191 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:20.113312 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:20.142978 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:20.173609 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:20.206223 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:20.259457 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:20.277187 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:20.287429 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:20.403373 29373 consensus_queue.cc:237] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:20.405833 29244 raft_consensus.cc:1261] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:20.406587 32362 raft_consensus.cc:1261] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:20.406549 4065 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:20.407277 4065 consensus_queue.cc:1035] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:20.408958 4065 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:20.409415 29244 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:20.412710 32362 raft_consensus.cc:2932] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:20.417093 29100 catalog_manager.cc:5484] T eb3c16421adb477e821495e785849cc4 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:20.426486 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3c16421adb477e821495e785849cc4 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:20.436345 4811 tablet_replica.cc:330] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:20.436650 4811 raft_consensus.cc:2229] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:20.437000 4811 raft_consensus.cc:2258] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:20.437997 4811 ts_tablet_manager.cc:1875] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:20.443464 4811 ts_tablet_manager.cc:1888] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:20.443578 4811 log.cc:1196] T eb3c16421adb477e821495e785849cc4 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb3c16421adb477e821495e785849cc4
I20240501 14:01:20.446854 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet eb3c16421adb477e821495e785849cc4 (table rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]) successfully deleted
I20240501 14:01:20.755911 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:20.760210 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:20.788661 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:20.811252 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:20.853492 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:20.859822 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:20.906728 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:20.979151 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:21.106374 32362 consensus_queue.cc:237] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:21.108219 31964 raft_consensus.cc:1261] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:21.108264 29373 raft_consensus.cc:1261] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:21.108850 4486 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:21.109155 4180 consensus_queue.cc:1035] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:21.112361 4180 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:21.112988 31963 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:21.117030 29373 raft_consensus.cc:2932] T 9c7c0ef199394de490b790a26532df3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:21.118669 29098 catalog_manager.cc:5484] T 9c7c0ef199394de490b790a26532df3b P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:21.138329 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7c0ef199394de490b790a26532df3b with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:21.138989 4909 tablet_replica.cc:330] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:21.139230 4909 raft_consensus.cc:2229] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:21.139498 4909 raft_consensus.cc:2258] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:21.140143 4909 ts_tablet_manager.cc:1875] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:21.145287 4909 ts_tablet_manager.cc:1888] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:21.145483 4909 log.cc:1196] T 9c7c0ef199394de490b790a26532df3b P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c7c0ef199394de490b790a26532df3b
I20240501 14:01:21.147054 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 9c7c0ef199394de490b790a26532df3b (table rebalancer_extra_table_44 [id=000bb64661ca4dbc8fa17d867ba3da55]) successfully deleted
I20240501 14:01:21.380502 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:21.425875 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:21.432284 32207 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:21.435891 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:21.491648 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:21.501399 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:21.522289 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:21.618096 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:21.699499 32222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d"
dest_uuid: "6a446ac802a7457d9f27076781f21d79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59954
I20240501 14:01:21.699738 32222 raft_consensus.cc:595] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:21.807248 4997 raft_consensus.cc:979] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: : Instructing follower 6edd6a3430d44c38a19c2804e555059f to start an election
I20240501 14:01:21.808243 4876 raft_consensus.cc:1067] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Signalling peer 6edd6a3430d44c38a19c2804e555059f to start an election
I20240501 14:01:21.811795 32361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d"
dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
from {username='slave'} at 127.22.33.71:34873
I20240501 14:01:21.812080 32361 raft_consensus.cc:482] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:21.812178 32361 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.815938 32361 raft_consensus.cc:504] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:21.817467 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:21.817646 32222 raft_consensus.cc:3032] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:21.817710 32222 raft_consensus.cc:729] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:01:21.817955 32222 consensus_queue.cc:260] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:21.818354 32222 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.820622 32222 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:01:21.822209 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:21.822304 32361 leader_election.cc:288] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:21.822417 29636 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.824582 29636 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:01:21.825039 32294 leader_election.cc:302] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:21.825206 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbb60619c1a4d8e8cd6266596a0614d" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:21.825341 4180 raft_consensus.cc:2781] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:21.825388 29507 raft_consensus.cc:3037] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.829063 29507 raft_consensus.cc:2454] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:01:21.843750 4180 raft_consensus.cc:686] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 2 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:21.845134 4180 consensus_queue.cc:237] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:21.850178 29098 catalog_manager.cc:5484] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 1 to 2, leader changed from 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 2 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:22.063249 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:22.067633 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:22.075758 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:22.097977 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:22.189630 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:22.200577 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:22.218928 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:01:22.287258 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 108479us
I20240501 14:01:22.313097 32222 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:22.313668 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:22.313752 4180 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:22.319154 29507 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:22.320068 4180 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:22.346220 29636 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:22.347033 4180 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:22.706095 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:22.711505 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:22.720057 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:22.744325 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:22.756052 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:22.759671 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:22.761226 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:22.809329 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:22.960827 29505 consensus_queue.cc:237] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:22.963099 29244 raft_consensus.cc:1261] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:22.963155 29373 raft_consensus.cc:1261] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:22.963744 5042 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:22.964056 4703 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:22.967012 5042 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:22.968243 29373 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:22.973224 29104 catalog_manager.cc:5484] T e190940ae8b44ca2bf1ee496a9d624ba P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:22.973446 29244 raft_consensus.cc:2932] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:22.985200 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet e190940ae8b44ca2bf1ee496a9d624ba with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:22.990331 5147 tablet_replica.cc:330] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:22.990511 5147 raft_consensus.cc:2229] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.990739 5147 raft_consensus.cc:2258] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.991333 5147 ts_tablet_manager.cc:1875] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:22.997968 5147 ts_tablet_manager.cc:1888] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:22.998075 5147 log.cc:1196] T e190940ae8b44ca2bf1ee496a9d624ba P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/e190940ae8b44ca2bf1ee496a9d624ba
I20240501 14:01:22.998616 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet e190940ae8b44ca2bf1ee496a9d624ba (table rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]) successfully deleted
I20240501 14:01:23.275770 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:23.309276 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:23.335867 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:23.363237 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:23.405803 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:01:23.434553 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 44009.3us
I20240501 14:01:23.436899 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:23.473459 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:23.492655 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:23.630708 29505 consensus_queue.cc:237] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:23.632619 29373 raft_consensus.cc:1261] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:23.632634 32223 raft_consensus.cc:1261] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:23.633188 4703 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:23.633482 5042 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:23.636698 5229 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:23.637181 29373 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:23.641898 32223 raft_consensus.cc:2932] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:23.644171 29099 catalog_manager.cc:5484] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:23.663829 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 8427dee24ad04247b23b24570c73c92a with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:23.665696 5238 tablet_replica.cc:330] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:23.665947 5238 raft_consensus.cc:2229] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:23.666209 5238 raft_consensus.cc:2258] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:23.666846 5238 ts_tablet_manager.cc:1875] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:23.671685 5238 ts_tablet_manager.cc:1888] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:23.671795 5238 log.cc:1196] T 8427dee24ad04247b23b24570c73c92a P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/8427dee24ad04247b23b24570c73c92a
I20240501 14:01:23.672453 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 8427dee24ad04247b23b24570c73c92a (table rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]) successfully deleted
I20240501 14:01:23.950886 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:23.957875 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:24.041702 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:24.044144 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:24.089574 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:24.098023 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:24.105679 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:24.150779 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:24.307958 32097 consensus_queue.cc:237] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:24.309937 29373 raft_consensus.cc:1261] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:24.309979 32222 raft_consensus.cc:1261] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:24.310554 4087 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:24.310806 5206 consensus_queue.cc:1035] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:24.313534 5206 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:24.314559 32223 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:24.317770 29373 raft_consensus.cc:2932] T 62c84a393bd047d985927417f82bec7d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:24.320184 29098 catalog_manager.cc:5484] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:24.326148 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c84a393bd047d985927417f82bec7d with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:24.329579 5315 tablet_replica.cc:330] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:24.329869 5315 raft_consensus.cc:2229] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:24.330091 5315 raft_consensus.cc:2258] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:24.330785 5315 ts_tablet_manager.cc:1875] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:24.338992 5315 ts_tablet_manager.cc:1888] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:24.339097 5315 log.cc:1196] T 62c84a393bd047d985927417f82bec7d P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/62c84a393bd047d985927417f82bec7d
I20240501 14:01:24.339874 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 62c84a393bd047d985927417f82bec7d (table rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]) successfully deleted
I20240501 14:01:24.700627 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:24.734699 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:24.833634 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:24.837580 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:24.847090 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:24.855067 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:24.935882 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:24.939050 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:25.085845 29246 consensus_queue.cc:237] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:25.088347 29377 raft_consensus.cc:1261] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:25.088378 32361 raft_consensus.cc:1261] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:25.088920 5121 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:25.089200 5209 consensus_queue.cc:1035] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:25.092082 5121 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.096020 29377 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.096858 32361 raft_consensus.cc:2932] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.103123 29097 catalog_manager.cc:5484] T ca0a888dc37d44c69dea73f9ffd556f6 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.110651 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0a888dc37d44c69dea73f9ffd556f6 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:25.111243 5408 tablet_replica.cc:330] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:25.113188 5408 raft_consensus.cc:2229] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:25.113641 5408 raft_consensus.cc:2258] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:25.114416 5408 ts_tablet_manager.cc:1875] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:25.120268 5408 ts_tablet_manager.cc:1888] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:25.120398 5408 log.cc:1196] T ca0a888dc37d44c69dea73f9ffd556f6 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca0a888dc37d44c69dea73f9ffd556f6
I20240501 14:01:25.121191 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet ca0a888dc37d44c69dea73f9ffd556f6 (table rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]) successfully deleted
I20240501 14:01:25.438822 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:25.449892 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:25.465529 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:25.525898 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:25.559816 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:25.567957 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:25.571249 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:25.579778 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:25.746712 29637 consensus_queue.cc:237] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:25.748786 32362 raft_consensus.cc:1261] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:25.749936 5347 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:25.751281 29373 raft_consensus.cc:1261] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:25.751843 5347 consensus_queue.cc:1035] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:25.754314 5347 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.754902 32362 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.760162 29101 catalog_manager.cc:5484] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.760329 29373 raft_consensus.cc:2932] T 241a2fdfe954476ba9bd2cfeb4b931ac P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:25.780890 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 241a2fdfe954476ba9bd2cfeb4b931ac with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:25.781545 5502 tablet_replica.cc:330] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:25.781783 5502 raft_consensus.cc:2229] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:25.782075 5502 raft_consensus.cc:2258] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:25.782869 5502 ts_tablet_manager.cc:1875] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:25.789680 5502 ts_tablet_manager.cc:1888] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:25.789875 5502 log.cc:1196] T 241a2fdfe954476ba9bd2cfeb4b931ac P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/241a2fdfe954476ba9bd2cfeb4b931ac
I20240501 14:01:25.790663 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 241a2fdfe954476ba9bd2cfeb4b931ac (table rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]) successfully deleted
I20240501 14:01:26.146934 29224 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:26.166910 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:26.174760 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:26.179841 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:26.188728 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:26.229446 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:26.273860 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:26.308130 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:01:26.334791 29442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 45345.3us
I20240501 14:01:26.382005 29246 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba20c01d741a414196972ced1601baf5"
dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40800
I20240501 14:01:26.382298 29246 raft_consensus.cc:595] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:26.392910 5566 raft_consensus.cc:979] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: : Instructing follower ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:01:26.393123 5566 raft_consensus.cc:1067] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Signalling peer ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:01:26.394057 29637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba20c01d741a414196972ced1601baf5"
dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
from {username='slave'} at 127.22.33.65:53443
I20240501 14:01:26.394321 29637 raft_consensus.cc:482] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:26.394429 29637 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:26.396685 29637 raft_consensus.cc:504] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:26.397922 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:01:26.398026 29637 leader_election.cc:288] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:26.398111 29246 raft_consensus.cc:3032] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:26.398175 29246 raft_consensus.cc:729] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:01:26.398451 29246 consensus_queue.cc:260] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:26.398679 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:01:26.398869 32095 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:26.398885 29246 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:26.401804 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba20c01d741a414196972ced1601baf5" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:26.402000 29507 raft_consensus.cc:3037] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:26.402361 32095 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:26.404048 29507 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:26.404428 29571 leader_election.cc:302] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:26.404697 4439 raft_consensus.cc:2781] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:26.404817 4439 raft_consensus.cc:686] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:26.405076 4439 consensus_queue.cc:237] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:26.409102 29246 raft_consensus.cc:2454] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:26.412045 29103 catalog_manager.cc:5484] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 1 to 2, leader changed from 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:26.736068 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:26.736291 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:26.745779 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:26.773996 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:26.818498 29246 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:26.819809 5529 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:26.823416 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:26.834244 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:26.893069 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:26.904245 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:26.936957 32095 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:26.938136 5529 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:26.990928 29507 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:26.991775 5529 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:27.351806 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:27.386451 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:27.390700 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:27.391021 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:27.391017 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:27.441195 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:27.479393 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:27.527101 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:27.649233 32095 consensus_queue.cc:237] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:27.651844 29374 raft_consensus.cc:1261] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:27.651877 29637 raft_consensus.cc:1261] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:27.652401 4087 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:27.652755 5642 consensus_queue.cc:1035] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:27.655627 4087 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:27.656297 29637 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:27.657181 29374 raft_consensus.cc:2932] T 0ff2fc8d230a45daa4e0c8ba4fec422e P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:27.664772 29101 catalog_manager.cc:5484] T 0ff2fc8d230a45daa4e0c8ba4fec422e P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:27.672154 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ff2fc8d230a45daa4e0c8ba4fec422e with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:27.672704 5746 tablet_replica.cc:330] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:27.672894 5746 raft_consensus.cc:2229] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:27.673095 5746 raft_consensus.cc:2258] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:27.673642 5746 ts_tablet_manager.cc:1875] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:27.678195 5746 ts_tablet_manager.cc:1888] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:27.678287 5746 log.cc:1196] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/0ff2fc8d230a45daa4e0c8ba4fec422e
I20240501 14:01:27.679705 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 0ff2fc8d230a45daa4e0c8ba4fec422e (table rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]) successfully deleted
I20240501 14:01:27.981300 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:28.013258 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:28.040074 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:28.078775 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:28.101223 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:28.103462 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:28.224143 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:28.241297 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:28.337390 31965 consensus_queue.cc:237] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:28.339339 32361 raft_consensus.cc:1261] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:28.339408 32223 raft_consensus.cc:1261] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:28.339887 5257 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:28.340158 5569 consensus_queue.cc:1035] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:28.343592 5569 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:28.346030 32223 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:28.352267 29104 catalog_manager.cc:5484] T 59c54188ad9e40dd925b50a250dcea04 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:28.355402 32361 raft_consensus.cc:2932] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:28.364811 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c54188ad9e40dd925b50a250dcea04 with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:28.373991 5832 tablet_replica.cc:330] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:28.374261 5832 raft_consensus.cc:2229] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:28.374588 5832 raft_consensus.cc:2258] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:28.375355 5832 ts_tablet_manager.cc:1875] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:28.382341 5832 ts_tablet_manager.cc:1888] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:28.382508 5832 log.cc:1196] T 59c54188ad9e40dd925b50a250dcea04 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/59c54188ad9e40dd925b50a250dcea04
I20240501 14:01:28.383312 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 59c54188ad9e40dd925b50a250dcea04 (table rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]) successfully deleted
I20240501 14:01:28.772214 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:28.792338 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:28.806702 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:28.812007 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:28.825276 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:28.873549 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:28.874964 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:01:28.902978 29181 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 106143us
I20240501 14:01:28.904249 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:29.116139 32359 consensus_queue.cc:237] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:29.118544 29374 raft_consensus.cc:1261] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:29.118930 29246 raft_consensus.cc:1261] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:29.119110 4999 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:29.119488 5843 consensus_queue.cc:1035] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:29.121898 4999 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:29.122563 29246 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:29.122570 29374 raft_consensus.cc:2932] T e83b7b4decde4e459e49de9eb05b8c0f P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:29.129213 29105 catalog_manager.cc:5484] T e83b7b4decde4e459e49de9eb05b8c0f P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:29.160388 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet e83b7b4decde4e459e49de9eb05b8c0f with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:29.171551 5926 tablet_replica.cc:330] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:29.171792 5926 raft_consensus.cc:2229] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:29.172122 5926 raft_consensus.cc:2258] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:29.172832 5926 ts_tablet_manager.cc:1875] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:29.181079 5926 ts_tablet_manager.cc:1888] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:29.181239 5926 log.cc:1196] T e83b7b4decde4e459e49de9eb05b8c0f P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/e83b7b4decde4e459e49de9eb05b8c0f
I20240501 14:01:29.182261 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet e83b7b4decde4e459e49de9eb05b8c0f (table rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]) successfully deleted
I20240501 14:01:29.435256 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:29.442075 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:29.495771 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:29.560920 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:29.617478 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:29.636436 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:29.689648 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:29.699294 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:29.816217 29636 consensus_queue.cc:237] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:29.822459 32360 raft_consensus.cc:1261] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:29.823081 29246 raft_consensus.cc:1261] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:29.823570 5925 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:29.823917 5651 consensus_queue.cc:1035] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:29.826784 5925 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:29.827394 29244 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:29.829267 32360 raft_consensus.cc:2932] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:29.836315 29097 catalog_manager.cc:5484] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:29.870710 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 798c9c25344d4beb9a079fb776044a17 with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:29.872644 6005 tablet_replica.cc:330] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:29.872952 6005 raft_consensus.cc:2229] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:29.873715 6005 raft_consensus.cc:2258] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:29.875366 6005 ts_tablet_manager.cc:1875] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:29.884159 6005 ts_tablet_manager.cc:1888] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:29.884403 6005 log.cc:1196] T 798c9c25344d4beb9a079fb776044a17 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/798c9c25344d4beb9a079fb776044a17
I20240501 14:01:29.886137 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 798c9c25344d4beb9a079fb776044a17 (table rebalancer_extra_table_50 [id=a9fb113d54a943e29371eb576da18c6f]) successfully deleted
I20240501 14:01:30.200958 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:30.207563 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:30.226157 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:30.250041 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:30.290977 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:30.312718 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:30.332736 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:30.390730 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:30.511889 31964 consensus_queue.cc:237] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:30.517988 32222 raft_consensus.cc:1261] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:30.518043 29372 raft_consensus.cc:1261] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:30.518497 5257 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:30.519050 5569 consensus_queue.cc:1035] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:30.523205 5257 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:30.524207 29372 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:30.528534 32223 raft_consensus.cc:2932] T bfbf43ae81ae45ba8d5abd345d18ac8a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:30.532207 29099 catalog_manager.cc:5484] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:30.557022 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet bfbf43ae81ae45ba8d5abd345d18ac8a with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:30.558228 6094 tablet_replica.cc:330] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:30.558517 6094 raft_consensus.cc:2229] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:30.558733 6094 raft_consensus.cc:2258] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:30.559319 6094 ts_tablet_manager.cc:1875] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:30.566041 6094 ts_tablet_manager.cc:1888] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:30.566222 6094 log.cc:1196] T bfbf43ae81ae45ba8d5abd345d18ac8a P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/bfbf43ae81ae45ba8d5abd345d18ac8a
I20240501 14:01:30.567016 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet bfbf43ae81ae45ba8d5abd345d18ac8a (table rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]) successfully deleted
I20240501 14:01:30.844724 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:30.845233 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:30.853488 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:30.858381 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:30.874861 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:30.880417 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:30.887871 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:30.952529 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:31.127840 29636 consensus_queue.cc:237] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:31.129894 32222 raft_consensus.cc:1261] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:31.129999 29377 raft_consensus.cc:1261] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:31.130729 5651 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:31.131004 5925 consensus_queue.cc:1035] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:31.133527 5925 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:31.134377 29377 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:31.140079 32224 raft_consensus.cc:2932] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:31.143378 29098 catalog_manager.cc:5484] T 34d3876037334884a4a36bc192956cf4 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:31.155767 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d3876037334884a4a36bc192956cf4 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:31.156900 6176 tablet_replica.cc:330] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:31.157199 6176 raft_consensus.cc:2229] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:31.157476 6176 raft_consensus.cc:2258] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:31.158236 6176 ts_tablet_manager.cc:1875] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:31.168133 6176 ts_tablet_manager.cc:1888] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:31.168372 6176 log.cc:1196] T 34d3876037334884a4a36bc192956cf4 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/34d3876037334884a4a36bc192956cf4
I20240501 14:01:31.169423 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 34d3876037334884a4a36bc192956cf4 (table rebalancer_extra_table_38 [id=47c80c186884434d8ade8ed82e71ffcb]) successfully deleted
I20240501 14:01:31.240099 29505 consensus_queue.cc:237] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:01:31.244237 29442 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f3d8b11a9bd4f69a855aa087e7cbb5d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:31.245301 29244 raft_consensus.cc:1261] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:31.246443 6204 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:31.248762 32224 raft_consensus.cc:1261] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:31.249354 5570 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:31.252111 6205 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:31.253314 32224 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:31.259747 29097 catalog_manager.cc:5484] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:31.265386 29244 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:31.299989 29377 consensus_queue.cc:237] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
W20240501 14:01:31.303221 29312 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0efddc01769f43e0b6f44d40ce6fa562. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:31.303324 29505 raft_consensus.cc:1261] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:31.303498 29636 raft_consensus.cc:1261] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:31.303953 5034 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:31.304188 5787 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:31.322353 6225 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:31.323057 29505 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:31.328956 29097 catalog_manager.cc:5484] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:31.337164 29636 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:31.658555 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:31.682498 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:31.688443 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:31.730563 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:31.754601 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:31.763585 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:31.764000 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:31.773213 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:31.869488 6292 ts_tablet_manager.cc:902] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:31.933283 6292 tablet_copy_client.cc:308] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:31.934242 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:54045)
I20240501 14:01:31.934404 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:54045: session id = 9903ed35bca144bcb74e9bf5102117e7-3f3d8b11a9bd4f69a855aa087e7cbb5d
I20240501 14:01:31.937366 29527 tablet_copy_source_session.cc:205] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:31.938861 6292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.944882 6296 ts_tablet_manager.cc:902] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:31.946350 6292 tablet_copy_client.cc:783] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:31.946628 6292 tablet_copy_client.cc:647] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:31.950562 6292 tablet_copy_client.cc:523] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:31.953625 6292 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:31.960367 6296 tablet_copy_client.cc:308] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:31.961215 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 0efddc01769f43e0b6f44d40ce6fa562 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:38381)
I20240501 14:01:31.961401 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 0efddc01769f43e0b6f44d40ce6fa562 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:38381: session id = 0a697f3a1ee544daa7073814ab845d58-0efddc01769f43e0b6f44d40ce6fa562
I20240501 14:01:31.964320 29397 tablet_copy_source_session.cc:205] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:31.967248 6296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0efddc01769f43e0b6f44d40ce6fa562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.971805 6292 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:31.972281 6292 tablet_bootstrap.cc:492] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:01:31.972568 6292 ts_tablet_manager.cc:1367] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20240501 14:01:31.972743 6296 tablet_copy_client.cc:783] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:31.973008 6296 tablet_copy_client.cc:647] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:31.973980 6292 raft_consensus.cc:348] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:31.974295 6292 raft_consensus.cc:729] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.974584 6292 consensus_queue.cc:260] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:31.975466 6292 ts_tablet_manager.cc:1398] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:31.976289 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-3f3d8b11a9bd4f69a855aa087e7cbb5d received from {username='slave'} at 127.22.33.69:54045
I20240501 14:01:31.976418 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-3f3d8b11a9bd4f69a855aa087e7cbb5d on tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:31.981608 6296 tablet_copy_client.cc:523] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:31.986546 6296 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:32.020622 29246 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:32.023203 32223 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:32.023212 29372 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:32.023820 5404 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:32.024340 5404 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:32.027168 5893 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:32.033607 32223 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:32.043489 29096 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:32.060004 6296 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:32.060606 6296 tablet_bootstrap.cc:492] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:32.060961 6296 ts_tablet_manager.cc:1367] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.074s user 0.042s sys 0.004s
I20240501 14:01:32.071123 6296 raft_consensus.cc:348] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:32.071414 6296 raft_consensus.cc:729] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.072535 29377 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:32.077206 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 500b9c9bdfa649bd883d6ef2d6e541fc with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
W20240501 14:01:32.077636 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 44036.2us
I20240501 14:01:32.080137 6296 consensus_queue.cc:260] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:32.081421 6332 tablet_replica.cc:330] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:32.081791 6332 raft_consensus.cc:2229] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:32.082178 6332 raft_consensus.cc:2258] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:32.083156 6332 ts_tablet_manager.cc:1875] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:32.085105 6296 ts_tablet_manager.cc:1398] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20240501 14:01:32.089533 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-0efddc01769f43e0b6f44d40ce6fa562 received from {username='slave'} at 127.22.33.70:38381
I20240501 14:01:32.089670 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-0efddc01769f43e0b6f44d40ce6fa562 on tablet 0efddc01769f43e0b6f44d40ce6fa562 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:32.092283 6332 ts_tablet_manager.cc:1888] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:32.092504 6332 log.cc:1196] T 500b9c9bdfa649bd883d6ef2d6e541fc P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/500b9c9bdfa649bd883d6ef2d6e541fc
I20240501 14:01:32.093552 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 500b9c9bdfa649bd883d6ef2d6e541fc (table rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]) successfully deleted
I20240501 14:01:32.327788 31964 raft_consensus.cc:1203] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:32.386297 32097 raft_consensus.cc:1203] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:32.420697 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:32.459407 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:32.493822 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:32.508303 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:32.512136 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:32.513808 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:32.548288 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:01:32.590166 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 56904.7us
I20240501 14:01:32.599059 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:32.766400 29377 consensus_queue.cc:237] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:32.768393 32097 raft_consensus.cc:1261] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:32.768826 29507 raft_consensus.cc:1261] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:32.768893 6225 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:32.769367 6225 consensus_queue.cc:1035] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:32.771346 5034 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:32.772047 32097 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:32.776858 29506 raft_consensus.cc:2932] T 0efddc01769f43e0b6f44d40ce6fa562 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:32.780444 29104 catalog_manager.cc:5484] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:32.804862 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 0efddc01769f43e0b6f44d40ce6fa562 with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52788
I20240501 14:01:32.805599 6424 tablet_replica.cc:330] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:01:32.805786 6424 raft_consensus.cc:2229] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:32.806084 6424 raft_consensus.cc:2258] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:32.806771 6424 ts_tablet_manager.cc:1875] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:32.812443 6424 ts_tablet_manager.cc:1888] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:32.813655 6424 log.cc:1196] T 0efddc01769f43e0b6f44d40ce6fa562 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/0efddc01769f43e0b6f44d40ce6fa562
I20240501 14:01:32.815230 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet 0efddc01769f43e0b6f44d40ce6fa562 (table rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]) successfully deleted
I20240501 14:01:33.120563 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:33.130581 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:33.161971 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:33.167656 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:33.173120 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:33.248126 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:33.261235 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:33.289933 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:33.425801 32360 consensus_queue.cc:237] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:33.428380 29506 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:33.428638 29636 raft_consensus.cc:1261] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:33.428995 6291 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:33.429302 5980 consensus_queue.cc:1035] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:33.433287 5980 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:33.448040 29506 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:33.450353 29101 catalog_manager.cc:5484] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 2 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:33.473505 29636 raft_consensus.cc:2932] T 6fbb60619c1a4d8e8cd6266596a0614d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:33.488252 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbb60619c1a4d8e8cd6266596a0614d with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:33.503286 6520 tablet_replica.cc:330] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:33.503532 6520 raft_consensus.cc:2229] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:33.503731 6520 raft_consensus.cc:2258] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:33.504415 6520 ts_tablet_manager.cc:1875] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:33.510862 6520 ts_tablet_manager.cc:1888] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 14:01:33.511091 6520 log.cc:1196] T 6fbb60619c1a4d8e8cd6266596a0614d P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/6fbb60619c1a4d8e8cd6266596a0614d
I20240501 14:01:33.512045 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 6fbb60619c1a4d8e8cd6266596a0614d (table rebalancer_extra_table_10 [id=41a2440f6b5d4795bcbd10f10dc80333]) successfully deleted
I20240501 14:01:33.875298 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:33.882140 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:33.915697 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:33.969285 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:34.007678 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:34.012387 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:34.084654 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:34.105067 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:34.248009 29507 consensus_queue.cc:237] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:34.250720 31964 raft_consensus.cc:1261] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:34.250954 32222 raft_consensus.cc:1261] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:34.251344 6205 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:34.251638 6482 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:34.254698 6482 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:34.255705 32222 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:34.257659 31964 raft_consensus.cc:2932] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:34.262624 29104 catalog_manager.cc:5484] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:34.286845 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:34.287480 6621 tablet_replica.cc:330] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:34.287742 6621 raft_consensus.cc:2229] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:34.287967 6621 raft_consensus.cc:2258] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:34.288686 6621 ts_tablet_manager.cc:1875] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:34.296181 6621 ts_tablet_manager.cc:1888] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:34.296339 6621 log.cc:1196] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f3d8b11a9bd4f69a855aa087e7cbb5d
I20240501 14:01:34.297107 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d (table rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]) successfully deleted
I20240501 14:01:34.688732 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:34.715104 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:34.736765 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:34.766875 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:34.771034 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:34.810194 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:34.810596 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:34.827965 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:34.981946 29637 consensus_queue.cc:237] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:34.984105 32097 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:34.984383 29506 raft_consensus.cc:1261] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:34.984774 6172 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:34.985263 6668 consensus_queue.cc:1035] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:34.988104 6668 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:34.988811 32097 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:34.989171 29506 raft_consensus.cc:2932] T ba20c01d741a414196972ced1601baf5 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:34.994663 29105 catalog_manager.cc:5484] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 6, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.000895 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet ba20c01d741a414196972ced1601baf5 with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48910
I20240501 14:01:35.001569 6713 tablet_replica.cc:330] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:01:35.001837 6713 raft_consensus.cc:2229] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:35.002053 6713 raft_consensus.cc:2258] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:35.002794 6713 ts_tablet_manager.cc:1875] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:35.011579 6713 ts_tablet_manager.cc:1888] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:01:35.011754 6713 log.cc:1196] T ba20c01d741a414196972ced1601baf5 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba20c01d741a414196972ced1601baf5
I20240501 14:01:35.013221 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet ba20c01d741a414196972ced1601baf5 (table rebalance_test_table_17 [id=1e2b8a4427e44b4184a0cfcc7047d71a]) successfully deleted
I20240501 14:01:35.127792 32222 consensus_queue.cc:237] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:35.130457 31963 raft_consensus.cc:1261] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.130718 32163 consensus_peers.cc:485] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25a6e3129a2f4bf89a36cdca5eaeb376. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.131021 6760 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.131156 32359 raft_consensus.cc:1261] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.131619 6472 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.135243 6760 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:35.135933 32359 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:35.140688 31963 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:35.147822 29105 catalog_manager.cc:5484] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:35.207628 32359 consensus_queue.cc:237] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:35.212035 31963 raft_consensus.cc:1261] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.212088 32096 raft_consensus.cc:1261] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.212352 32294 consensus_peers.cc:485] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36b0222ebb6d4246ab8551d5579b1760. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.212690 6780 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.213140 6684 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.216910 6684 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.217720 31963 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.221932 32097 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.225878 29099 catalog_manager.cc:5484] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.278781 29506 consensus_queue.cc:237] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:35.280947 29377 raft_consensus.cc:1261] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:35.281275 29637 raft_consensus.cc:1261] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:35.281749 6789 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:35.282027 6790 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:35.287952 29441 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2e5851626b54e1caf6105a70ccafc43. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.291779 6789 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:35.292263 29637 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:35.300258 29105 catalog_manager.cc:5484] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:35.302737 29377 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:35.343588 29637 consensus_queue.cc:237] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:35.345639 32222 raft_consensus.cc:1261] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.345773 32097 raft_consensus.cc:1261] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.346061 29570 consensus_peers.cc:485] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ec9bb1d59ac4b11a1f9b0d44b1dc69b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.346174 6668 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.346529 5925 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.349903 6668 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.350481 32222 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.353655 32097 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.359926 29105 catalog_manager.cc:5484] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.398169 29244 consensus_queue.cc:237] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:35.400508 29506 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:35.400605 29377 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:35.401038 29181 consensus_peers.cc:485] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6973301bdd94c2b8f98e718715ff752. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.401345 5404 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:35.401777 5404 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:35.404278 6329 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.404955 29377 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.410292 29506 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.412281 29099 catalog_manager.cc:5484] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.472258 29244 consensus_queue.cc:237] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:35.475708 32362 raft_consensus.cc:1261] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.475777 31963 raft_consensus.cc:1261] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.476348 6842 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.476634 5404 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:35.480500 29179 consensus_peers.cc:485] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5ed5a5797424f0f88c2b3c711468e6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.506258 6842 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.507012 31963 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.510794 32362 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.515995 29097 catalog_manager.cc:5484] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:35.591063 32096 consensus_queue.cc:237] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:35.593318 29506 raft_consensus.cc:1261] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.593371 29377 raft_consensus.cc:1261] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.593485 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a38ce68ec4d4ba3a813bdb0ec314d4d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.593842 6497 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.594378 6863 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.596912 6497 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:35.597550 29506 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:35.604210 29377 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:35.606179 29101 catalog_manager.cc:5484] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:35.625334 6890 ts_tablet_manager.cc:902] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:01:35.626890 6890 tablet_copy_client.cc:308] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:01:35.627609 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet 25a6e3129a2f4bf89a36cdca5eaeb376 from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:56853)
I20240501 14:01:35.627763 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet 25a6e3129a2f4bf89a36cdca5eaeb376 from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:56853: session id = b87d188552494b4590afc37a85b98c35-25a6e3129a2f4bf89a36cdca5eaeb376
I20240501 14:01:35.631713 32248 tablet_copy_source_session.cc:205] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:35.635427 6890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6e3129a2f4bf89a36cdca5eaeb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.641237 6890 tablet_copy_client.cc:783] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:35.641479 6890 tablet_copy_client.cc:647] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:35.643249 6890 tablet_copy_client.cc:523] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:35.646965 6890 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:35.651414 32096 consensus_queue.cc:237] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:35.652644 6890 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:35.652999 6890 tablet_bootstrap.cc:492] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:35.653252 6890 ts_tablet_manager.cc:1367] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:01:35.653911 6890 raft_consensus.cc:348] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:35.654078 6890 raft_consensus.cc:729] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.654369 29636 raft_consensus.cc:1261] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.654347 6890 consensus_queue.cc:260] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:35.654870 6890 ts_tablet_manager.cc:1398] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:35.655138 32362 raft_consensus.cc:1261] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.655567 32030 consensus_peers.cc:485] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c0fc97541db4467b5089bbeaf9bfca0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.655843 6863 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.655902 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-25a6e3129a2f4bf89a36cdca5eaeb376 received from {username='slave'} at 127.22.33.66:56853
I20240501 14:01:35.656026 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session b87d188552494b4590afc37a85b98c35-25a6e3129a2f4bf89a36cdca5eaeb376 on tablet 25a6e3129a2f4bf89a36cdca5eaeb376 with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:35.657297 6863 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.674474 6909 ts_tablet_manager.cc:902] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:35.675246 6863 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.675877 29636 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.677608 6909 tablet_copy_client.cc:308] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:01:35.678591 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet 36b0222ebb6d4246ab8551d5579b1760 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:52747)
I20240501 14:01:35.678742 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet 36b0222ebb6d4246ab8551d5579b1760 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:52747: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-36b0222ebb6d4246ab8551d5579b1760
I20240501 14:01:35.681525 32383 tablet_copy_source_session.cc:205] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:35.684803 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b0222ebb6d4246ab8551d5579b1760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.685099 29099 catalog_manager.cc:5484] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.685554 32359 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:35.701467 6909 tablet_copy_client.cc:783] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:35.701738 6909 tablet_copy_client.cc:647] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:35.704114 6909 tablet_copy_client.cc:523] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:35.708451 6909 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:35.720404 6909 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:35.720883 6909 tablet_bootstrap.cc:492] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:35.721259 6909 ts_tablet_manager.cc:1367] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.007s
I20240501 14:01:35.722237 6909 raft_consensus.cc:348] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:35.722523 6909 raft_consensus.cc:729] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.722805 6909 consensus_queue.cc:260] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:35.724267 6909 ts_tablet_manager.cc:1398] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:01:35.725059 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-36b0222ebb6d4246ab8551d5579b1760 received from {username='slave'} at 127.22.33.68:52747
I20240501 14:01:35.725201 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-36b0222ebb6d4246ab8551d5579b1760 on tablet 36b0222ebb6d4246ab8551d5579b1760 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:35.754128 29377 consensus_queue.cc:237] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:35.756374 29507 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:35.756803 29244 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:35.757223 29312 consensus_peers.cc:485] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71933eccec5f4fdc8b1df7111f941777. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.757359 6932 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:35.757660 6419 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:35.760344 6419 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.762064 29244 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.765272 29507 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.771266 29101 catalog_manager.cc:5484] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:35.834113 29636 consensus_queue.cc:237] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:35.838039 31964 raft_consensus.cc:1261] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:35.838363 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e32147255254f2d8baff46c9b2d0e60. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:35.838344 32359 raft_consensus.cc:1261] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:35.838979 6668 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.839267 5925 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:35.863363 6955 ts_tablet_manager.cc:902] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:35.865525 6952 ts_tablet_manager.cc:902] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:35.870834 6955 tablet_copy_client.cc:308] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:35.871596 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet d2e5851626b54e1caf6105a70ccafc43 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:44367)
I20240501 14:01:35.871769 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet d2e5851626b54e1caf6105a70ccafc43 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:44367: session id = 6edd6a3430d44c38a19c2804e555059f-d2e5851626b54e1caf6105a70ccafc43
I20240501 14:01:35.872903 6668 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:35.874547 29527 tablet_copy_source_session.cc:205] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:35.878952 32362 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:35.886168 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e5851626b54e1caf6105a70ccafc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.894268 6909 ts_tablet_manager.cc:902] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:35.897606 29099 catalog_manager.cc:5484] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:35.903997 31964 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:35.913115 6955 tablet_copy_client.cc:783] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:35.913789 6955 tablet_copy_client.cc:647] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:35.917029 6955 tablet_copy_client.cc:523] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:35.920538 6955 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:35.934861 6952 tablet_copy_client.cc:308] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:35.935685 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet c6973301bdd94c2b8f98e718715ff752 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36699)
I20240501 14:01:35.935850 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet c6973301bdd94c2b8f98e718715ff752 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36699: session id = 0a697f3a1ee544daa7073814ab845d58-c6973301bdd94c2b8f98e718715ff752
I20240501 14:01:35.938853 29266 tablet_copy_source_session.cc:205] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:35.940376 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6973301bdd94c2b8f98e718715ff752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.942315 6909 tablet_copy_client.cc:308] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:35.943063 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet e5ed5a5797424f0f88c2b3c711468e6b from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:55329)
I20240501 14:01:35.943230 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet e5ed5a5797424f0f88c2b3c711468e6b from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:55329: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-e5ed5a5797424f0f88c2b3c711468e6b
I20240501 14:01:35.947557 29266 tablet_copy_source_session.cc:205] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:35.952852 6952 tablet_copy_client.cc:783] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:35.953503 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed5a5797424f0f88c2b3c711468e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.957845 6952 tablet_copy_client.cc:647] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:35.960322 6909 tablet_copy_client.cc:783] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:35.960543 6909 tablet_copy_client.cc:647] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:35.962741 6909 tablet_copy_client.cc:523] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:35.965642 6952 tablet_copy_client.cc:523] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:35.966140 6909 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:35.974689 6980 ts_tablet_manager.cc:902] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:35.987684 6952 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:35.999712 6909 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.000125 6909 tablet_bootstrap.cc:492] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:36.000371 6909 ts_tablet_manager.cc:1367] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.014s
I20240501 14:01:36.001140 6909 raft_consensus.cc:348] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.001446 6909 raft_consensus.cc:729] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.003376 6909 consensus_queue.cc:260] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.004213 6909 ts_tablet_manager.cc:1398] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:36.005019 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-e5ed5a5797424f0f88c2b3c711468e6b received from {username='slave'} at 127.22.33.68:55329
I20240501 14:01:36.005136 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-e5ed5a5797424f0f88c2b3c711468e6b on tablet e5ed5a5797424f0f88c2b3c711468e6b with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:36.017958 6980 tablet_copy_client.cc:308] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:36.018707 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:53443)
I20240501 14:01:36.018874 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:53443: session id = 6518b7a04854458bb2fb23edc8b6f7ba-7ec9bb1d59ac4b11a1f9b0d44b1dc69b
I20240501 14:01:36.021353 29657 tablet_copy_source_session.cc:205] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.022891 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.026281 29374 consensus_queue.cc:237] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.030833 29377 raft_consensus.cc:1203] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.035251 6980 tablet_copy_client.cc:783] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.035559 6980 tablet_copy_client.cc:647] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.038456 6980 tablet_copy_client.cc:523] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.039860 6955 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.040339 6955 tablet_bootstrap.cc:492] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:01:36.040642 6955 ts_tablet_manager.cc:1367] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.120s user 0.044s sys 0.012s
I20240501 14:01:36.041503 6955 raft_consensus.cc:348] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:36.041754 6955 raft_consensus.cc:729] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.042419 6955 consensus_queue.cc:260] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:36.044082 6980 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:36.044818 6955 ts_tablet_manager.cc:1398] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:01:36.046486 32359 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:36.046763 29310 consensus_peers.cc:485] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe99c587804f47c3b58ec3ddbab8a371. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:36.047255 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-d2e5851626b54e1caf6105a70ccafc43 received from {username='slave'} at 127.22.33.72:44367
I20240501 14:01:36.047374 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-d2e5851626b54e1caf6105a70ccafc43 on tablet d2e5851626b54e1caf6105a70ccafc43 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:36.051585 6419 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.069470 31964 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:36.070331 6419 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.073216 6993 ts_tablet_manager.cc:902] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:36.095638 6999 ts_tablet_manager.cc:902] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:36.097326 6419 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.102659 32360 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.107966 31963 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.111122 29101 catalog_manager.cc:5484] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.124759 6980 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.125245 6980 tablet_bootstrap.cc:492] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:36.125630 6980 ts_tablet_manager.cc:1367] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.082s user 0.022s sys 0.029s
I20240501 14:01:36.126719 6980 raft_consensus.cc:348] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:36.127378 6980 raft_consensus.cc:729] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.127693 6980 consensus_queue.cc:260] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:36.128286 6980 ts_tablet_manager.cc:1398] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:01:36.129318 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-7ec9bb1d59ac4b11a1f9b0d44b1dc69b received from {username='slave'} at 127.22.33.65:53443
I20240501 14:01:36.129455 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-7ec9bb1d59ac4b11a1f9b0d44b1dc69b on tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:36.130955 6993 tablet_copy_client.cc:308] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:36.131707 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:59037)
I20240501 14:01:36.131860 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:59037: session id = 6a446ac802a7457d9f27076781f21d79-4a38ce68ec4d4ba3a813bdb0ec314d4d
I20240501 14:01:36.134948 32117 tablet_copy_source_session.cc:205] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.138993 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.148563 29636 raft_consensus.cc:1203] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.155462 6952 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.156275 6952 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:36.156517 6952 ts_tablet_manager.cc:1367] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.169s user 0.055s sys 0.000s
I20240501 14:01:36.157284 6952 raft_consensus.cc:348] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:36.157490 6952 raft_consensus.cc:729] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.157668 6952 consensus_queue.cc:260] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:36.159715 6952 ts_tablet_manager.cc:1398] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:36.160351 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-c6973301bdd94c2b8f98e718715ff752 received from {username='slave'} at 127.22.33.70:36699
I20240501 14:01:36.160480 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-c6973301bdd94c2b8f98e718715ff752 on tablet c6973301bdd94c2b8f98e718715ff752 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:36.165457 6993 tablet_copy_client.cc:783] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.165721 6993 tablet_copy_client.cc:647] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.167553 6993 tablet_copy_client.cc:523] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.171427 6993 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:36.201560 6999 tablet_copy_client.cc:308] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:36.202646 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 5c0fc97541db4467b5089bbeaf9bfca0 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:54889)
I20240501 14:01:36.202836 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 5c0fc97541db4467b5089bbeaf9bfca0 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:54889: session id = 6518b7a04854458bb2fb23edc8b6f7ba-5c0fc97541db4467b5089bbeaf9bfca0
I20240501 14:01:36.205821 6993 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.206228 6993 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:01:36.206332 32117 tablet_copy_source_session.cc:205] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.206445 6993 ts_tablet_manager.cc:1367] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.011s
I20240501 14:01:36.207126 6993 raft_consensus.cc:348] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:36.207334 6993 raft_consensus.cc:729] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.208847 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0fc97541db4467b5089bbeaf9bfca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.207564 6993 consensus_queue.cc:260] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:36.210893 6993 ts_tablet_manager.cc:1398] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:01:36.211571 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-4a38ce68ec4d4ba3a813bdb0ec314d4d received from {username='slave'} at 127.22.33.71:59037
I20240501 14:01:36.211694 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-4a38ce68ec4d4ba3a813bdb0ec314d4d on tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:36.233572 6999 tablet_copy_client.cc:783] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.233992 6999 tablet_copy_client.cc:647] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.236580 6999 tablet_copy_client.cc:523] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.240377 6999 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:36.259389 6999 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.259773 6999 tablet_bootstrap.cc:492] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:36.260035 6999 ts_tablet_manager.cc:1367] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240501 14:01:36.260802 6999 raft_consensus.cc:348] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:36.261056 6999 raft_consensus.cc:729] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.261315 6999 consensus_queue.cc:260] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:36.262045 6999 ts_tablet_manager.cc:1398] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:36.265416 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-5c0fc97541db4467b5089bbeaf9bfca0 received from {username='slave'} at 127.22.33.65:54889
I20240501 14:01:36.265555 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-5c0fc97541db4467b5089bbeaf9bfca0 on tablet 5c0fc97541db4467b5089bbeaf9bfca0 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:36.291782 29246 consensus_queue.cc:237] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
W20240501 14:01:36.294773 29181 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e67dc79a086048a8839ae386857652ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:36.294894 32222 raft_consensus.cc:1261] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:36.294864 29506 raft_consensus.cc:1261] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:36.295660 6842 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.296705 6329 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.298429 6952 ts_tablet_manager.cc:902] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:36.307758 6842 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:36.308447 32222 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:36.312901 29506 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:36.320395 29101 catalog_manager.cc:5484] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:36.330767 6952 tablet_copy_client.cc:308] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:36.331498 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 71933eccec5f4fdc8b1df7111f941777 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:38381)
I20240501 14:01:36.331643 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 71933eccec5f4fdc8b1df7111f941777 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:38381: session id = 0a697f3a1ee544daa7073814ab845d58-71933eccec5f4fdc8b1df7111f941777
I20240501 14:01:36.335055 29397 tablet_copy_source_session.cc:205] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.336872 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71933eccec5f4fdc8b1df7111f941777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.337101 7034 ts_tablet_manager.cc:902] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:36.346211 6952 tablet_copy_client.cc:783] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.346681 6952 tablet_copy_client.cc:647] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.348659 29637 raft_consensus.cc:1203] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.349056 6952 tablet_copy_client.cc:523] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.353790 6952 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:36.358579 32096 raft_consensus.cc:1203] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:36.385362 7034 tablet_copy_client.cc:308] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:36.386109 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 9e32147255254f2d8baff46c9b2d0e60 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:47041)
I20240501 14:01:36.386296 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 9e32147255254f2d8baff46c9b2d0e60 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:47041: session id = 522f4ff5e9084cbf8462196533c7d7aa-9e32147255254f2d8baff46c9b2d0e60
I20240501 14:01:36.388870 29657 tablet_copy_source_session.cc:205] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.391254 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e32147255254f2d8baff46c9b2d0e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.400745 7034 tablet_copy_client.cc:783] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.401466 7034 tablet_copy_client.cc:647] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.403364 7034 tablet_copy_client.cc:523] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.408695 7034 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:36.414461 7034 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.415678 7034 tablet_bootstrap.cc:492] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:01:36.416396 7034 ts_tablet_manager.cc:1367] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:01:36.418433 7034 raft_consensus.cc:348] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:36.418762 7034 raft_consensus.cc:729] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.419152 7034 consensus_queue.cc:260] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:36.420138 7034 ts_tablet_manager.cc:1398] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:36.420792 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-9e32147255254f2d8baff46c9b2d0e60 received from {username='slave'} at 127.22.33.67:47041
I20240501 14:01:36.420892 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-9e32147255254f2d8baff46c9b2d0e60 on tablet 9e32147255254f2d8baff46c9b2d0e60 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:01:36.421887 6952 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.422294 6952 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:36.422264 32360 raft_consensus.cc:1203] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:36.422538 6952 ts_tablet_manager.cc:1367] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.069s user 0.053s sys 0.003s
I20240501 14:01:36.423341 6952 raft_consensus.cc:348] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:36.423581 6952 raft_consensus.cc:729] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.423822 6952 consensus_queue.cc:260] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:36.424537 6952 ts_tablet_manager.cc:1398] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:36.432648 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-71933eccec5f4fdc8b1df7111f941777 received from {username='slave'} at 127.22.33.70:38381
I20240501 14:01:36.432790 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-71933eccec5f4fdc8b1df7111f941777 on tablet 71933eccec5f4fdc8b1df7111f941777 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:36.445976 29637 consensus_queue.cc:237] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:36.449134 32222 raft_consensus.cc:1203] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.454831 32096 raft_consensus.cc:1261] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:36.455207 31963 raft_consensus.cc:1261] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:36.455806 29570 consensus_peers.cc:485] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab3c1d804ba74db794bc0abe68bc9f96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:36.456209 5925 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.456784 5925 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.472532 6668 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:36.472999 32097 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:36.479653 29099 catalog_manager.cc:5484] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:36.481531 31963 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:36.530267 29246 raft_consensus.cc:1203] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.547188 7066 ts_tablet_manager.cc:902] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:36.561285 7066 tablet_copy_client.cc:308] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:36.561959 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet fe99c587804f47c3b58ec3ddbab8a371 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:35721)
I20240501 14:01:36.562101 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet fe99c587804f47c3b58ec3ddbab8a371 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:35721: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-fe99c587804f47c3b58ec3ddbab8a371
I20240501 14:01:36.564954 29397 tablet_copy_source_session.cc:205] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.567128 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe99c587804f47c3b58ec3ddbab8a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.576648 7066 tablet_copy_client.cc:783] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.576933 7066 tablet_copy_client.cc:647] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.579015 7066 tablet_copy_client.cc:523] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.582206 7066 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:36.587360 32359 consensus_queue.cc:237] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.589579 7066 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.589984 7066 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:36.590230 7066 ts_tablet_manager.cc:1367] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:01:36.590320 32222 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:36.590665 31963 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:36.590696 32294 consensus_peers.cc:485] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5aa71ecef4d419eb36c7777790e6f08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:36.590972 6950 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.591424 6780 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:36.590947 7066 raft_consensus.cc:348] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.591887 7066 raft_consensus.cc:729] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.592161 7066 consensus_queue.cc:260] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:36.597218 7066 ts_tablet_manager.cc:1398] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240501 14:01:36.598152 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-fe99c587804f47c3b58ec3ddbab8a371 received from {username='slave'} at 127.22.33.68:35721
I20240501 14:01:36.598254 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-fe99c587804f47c3b58ec3ddbab8a371 on tablet fe99c587804f47c3b58ec3ddbab8a371 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:36.606247 6780 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.607000 31963 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.616458 29097 catalog_manager.cc:5484] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.620819 32222 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:36.679318 29244 raft_consensus.cc:1203] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.713363 7111 ts_tablet_manager.cc:902] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:36.732797 7111 tablet_copy_client.cc:308] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:36.734903 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet e67dc79a086048a8839ae386857652ea from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:52231)
I20240501 14:01:36.735080 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet e67dc79a086048a8839ae386857652ea from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:52231: session id = b87d188552494b4590afc37a85b98c35-e67dc79a086048a8839ae386857652ea
I20240501 14:01:36.737823 29266 tablet_copy_source_session.cc:205] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:36.739472 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67dc79a086048a8839ae386857652ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.747767 7111 tablet_copy_client.cc:783] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:36.748275 7111 tablet_copy_client.cc:647] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:36.753424 7111 tablet_copy_client.cc:523] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:36.756762 7111 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:36.762329 7111 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:36.762730 7111 tablet_bootstrap.cc:492] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:36.762981 7111 ts_tablet_manager.cc:1367] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:01:36.763609 7111 raft_consensus.cc:348] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:36.763831 7111 raft_consensus.cc:729] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.764384 7111 consensus_queue.cc:260] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:36.765079 7111 ts_tablet_manager.cc:1398] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:36.765733 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-e67dc79a086048a8839ae386857652ea received from {username='slave'} at 127.22.33.66:52231
I20240501 14:01:36.765841 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session b87d188552494b4590afc37a85b98c35-e67dc79a086048a8839ae386857652ea on tablet e67dc79a086048a8839ae386857652ea with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:36.818285 32097 raft_consensus.cc:1203] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:36.902323 29637 raft_consensus.cc:1203] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.905416 29506 raft_consensus.cc:1203] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:36.992942 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:37.002566 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:37.006121 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:37.015625 7156 ts_tablet_manager.cc:902] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:37.016008 7066 ts_tablet_manager.cc:902] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:37.023563 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:37.031812 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:37.055549 7156 tablet_copy_client.cc:308] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:37.058912 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet ab3c1d804ba74db794bc0abe68bc9f96 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:53443)
I20240501 14:01:37.059069 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet ab3c1d804ba74db794bc0abe68bc9f96 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:53443: session id = 6518b7a04854458bb2fb23edc8b6f7ba-ab3c1d804ba74db794bc0abe68bc9f96
I20240501 14:01:37.061916 29657 tablet_copy_source_session.cc:205] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:37.072031 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c1d804ba74db794bc0abe68bc9f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.079208 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:37.082522 7066 tablet_copy_client.cc:308] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:01:37.083238 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet b5aa71ecef4d419eb36c7777790e6f08 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:52747)
I20240501 14:01:37.083416 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet b5aa71ecef4d419eb36c7777790e6f08 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:52747: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-b5aa71ecef4d419eb36c7777790e6f08
I20240501 14:01:37.086627 32383 tablet_copy_source_session.cc:205] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:01:37.091787 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 62523.7us
I20240501 14:01:37.092264 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aa71ecef4d419eb36c7777790e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.112354 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:37.123247 7156 tablet_copy_client.cc:783] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:37.123521 7156 tablet_copy_client.cc:647] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:37.125627 7156 tablet_copy_client.cc:523] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:37.133332 7156 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:37.151319 7066 tablet_copy_client.cc:783] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:37.154098 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:37.157420 7066 tablet_copy_client.cc:647] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:37.184367 7156 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:37.184837 7156 tablet_bootstrap.cc:492] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:37.185031 7066 tablet_copy_client.cc:523] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:37.185127 7156 ts_tablet_manager.cc:1367] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.002s
I20240501 14:01:37.185781 7156 raft_consensus.cc:348] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:37.185997 7156 raft_consensus.cc:729] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.187142 7156 consensus_queue.cc:260] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:37.192168 7066 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:37.193576 7156 ts_tablet_manager.cc:1398] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:01:37.194429 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-ab3c1d804ba74db794bc0abe68bc9f96 received from {username='slave'} at 127.22.33.65:53443
I20240501 14:01:37.194555 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-ab3c1d804ba74db794bc0abe68bc9f96 on tablet ab3c1d804ba74db794bc0abe68bc9f96 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:37.229176 7066 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:37.229578 7066 tablet_bootstrap.cc:492] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:37.229826 7066 ts_tablet_manager.cc:1367] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.007s
I20240501 14:01:37.238308 7066 raft_consensus.cc:348] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:37.244686 7066 raft_consensus.cc:729] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.245213 7066 consensus_queue.cc:260] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:37.247007 7066 ts_tablet_manager.cc:1398] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240501 14:01:37.250445 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-b5aa71ecef4d419eb36c7777790e6f08 received from {username='slave'} at 127.22.33.68:52747
I20240501 14:01:37.250581 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-b5aa71ecef4d419eb36c7777790e6f08 on tablet b5aa71ecef4d419eb36c7777790e6f08 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:37.265744 29377 raft_consensus.cc:1203] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:37.518478 29311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 44875.9us
I20240501 14:01:37.536227 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:37.538662 29636 raft_consensus.cc:1203] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:37.545619 29244 raft_consensus.cc:1203] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:37.608599 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:37.608995 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:37.621898 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:37.641439 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:37.642928 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:37.675359 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:37.702000 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:38.159291 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:38.214625 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:38.223661 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:38.246035 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:38.303920 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:38.307639 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:38.326974 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:38.342808 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:38.500694 29377 consensus_queue.cc:237] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:38.503479 29636 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:38.503657 31964 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:38.504217 7218 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:38.504477 6419 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:38.508206 7218 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:38.509582 31964 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:38.523358 29097 catalog_manager.cc:5484] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:38.544003 29637 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:38.560745 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet fe99c587804f47c3b58ec3ddbab8a371 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:38.561473 7323 tablet_replica.cc:330] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:38.561704 7323 raft_consensus.cc:2229] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:38.562181 7323 raft_consensus.cc:2258] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:38.563237 7323 ts_tablet_manager.cc:1875] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:38.571014 7323 ts_tablet_manager.cc:1888] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:38.571144 7323 log.cc:1196] T fe99c587804f47c3b58ec3ddbab8a371 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/fe99c587804f47c3b58ec3ddbab8a371
I20240501 14:01:38.571846 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet fe99c587804f47c3b58ec3ddbab8a371 (table rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]) successfully deleted
I20240501 14:01:38.840544 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:38.849642 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:38.851686 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:38.916757 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:38.916787 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:38.947101 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:38.983564 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:39.035990 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:39.140671 29637 consensus_queue.cc:237] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:39.143160 29506 raft_consensus.cc:1261] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:39.143879 32359 raft_consensus.cc:1261] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:39.143986 6668 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:39.149614 6668 consensus_queue.cc:1035] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:39.151664 7278 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:39.152287 29506 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:39.153296 32359 raft_consensus.cc:2932] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:39.159631 29096 catalog_manager.cc:5484] T 9e32147255254f2d8baff46c9b2d0e60 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:39.205792 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e32147255254f2d8baff46c9b2d0e60 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:39.206368 7406 tablet_replica.cc:330] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:39.206562 7406 raft_consensus.cc:2229] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:39.206787 7406 raft_consensus.cc:2258] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:39.207468 7406 ts_tablet_manager.cc:1875] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:39.213478 7406 ts_tablet_manager.cc:1888] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:39.213657 7406 log.cc:1196] T 9e32147255254f2d8baff46c9b2d0e60 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/9e32147255254f2d8baff46c9b2d0e60
I20240501 14:01:39.214746 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 9e32147255254f2d8baff46c9b2d0e60 (table rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]) successfully deleted
I20240501 14:01:39.447543 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:39.469708 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:39.522715 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:39.556638 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:39.575904 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:39.592497 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:39.618695 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:39.623394 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:39.780530 32097 consensus_queue.cc:237] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:39.782732 29244 raft_consensus.cc:1261] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:39.782718 29636 raft_consensus.cc:1261] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:39.783293 7163 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:39.783587 7409 consensus_queue.cc:1035] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:39.786520 7409 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:39.787431 29244 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:39.794592 29105 catalog_manager.cc:5484] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:39.811630 29636 raft_consensus.cc:2932] T 5c0fc97541db4467b5089bbeaf9bfca0 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:39.834296 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c0fc97541db4467b5089bbeaf9bfca0 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:39.841802 7489 tablet_replica.cc:330] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:39.842065 7489 raft_consensus.cc:2229] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:39.842273 7489 raft_consensus.cc:2258] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:39.843055 7489 ts_tablet_manager.cc:1875] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:39.851864 7489 ts_tablet_manager.cc:1888] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:39.852130 7489 log.cc:1196] T 5c0fc97541db4467b5089bbeaf9bfca0 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/5c0fc97541db4467b5089bbeaf9bfca0
I20240501 14:01:39.852958 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 5c0fc97541db4467b5089bbeaf9bfca0 (table rebalancer_extra_table_22 [id=28fe00a56c2e4f1a88166fb5d1cf5628]) successfully deleted
I20240501 14:01:40.171341 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:40.234340 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:40.272812 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:40.334842 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:40.390189 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:40.396446 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:40.414479 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:40.499863 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:40.597793 32096 consensus_queue.cc:237] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:40.599702 29506 raft_consensus.cc:1261] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:40.599709 32224 raft_consensus.cc:1261] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:40.600239 7547 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:40.600515 7409 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:40.603510 7409 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:40.607206 29506 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:40.612025 32224 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:40.612543 29098 catalog_manager.cc:5484] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:40.630111 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51134
I20240501 14:01:40.631577 7575 tablet_replica.cc:330] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:01:40.632019 7575 raft_consensus.cc:2229] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:40.632375 7575 raft_consensus.cc:2258] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:40.633563 7575 ts_tablet_manager.cc:1875] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:40.639572 7575 ts_tablet_manager.cc:1888] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:40.639683 7575 log.cc:1196] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a38ce68ec4d4ba3a813bdb0ec314d4d
I20240501 14:01:40.640318 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d (table rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]) successfully deleted
I20240501 14:01:40.980407 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:40.991581 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:41.020946 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:41.097750 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:41.142473 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:41.216473 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:41.267367 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:41.267692 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:41.380703 29377 consensus_queue.cc:237] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:41.382707 29245 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:41.382752 32096 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:41.383250 6419 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:41.383508 7218 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:41.386523 7654 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:41.387471 29245 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:41.387869 32096 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:41.395788 29096 catalog_manager.cc:5484] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:41.408558 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 71933eccec5f4fdc8b1df7111f941777 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:41.413949 7664 tablet_replica.cc:330] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:41.414167 7664 raft_consensus.cc:2229] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:41.414439 7664 raft_consensus.cc:2258] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:41.415500 7664 ts_tablet_manager.cc:1875] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:41.422690 7664 ts_tablet_manager.cc:1888] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:41.422809 7664 log.cc:1196] T 71933eccec5f4fdc8b1df7111f941777 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/71933eccec5f4fdc8b1df7111f941777
I20240501 14:01:41.423610 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 71933eccec5f4fdc8b1df7111f941777 (table rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]) successfully deleted
I20240501 14:01:41.720407 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:41.753041 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:41.774446 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:41.784463 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:41.791064 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:41.811873 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:41.814112 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:41.822816 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:42.039526 29245 consensus_queue.cc:237] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:42.043081 29636 raft_consensus.cc:1261] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:42.043227 32360 raft_consensus.cc:1261] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:42.044198 6842 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:42.044845 6842 consensus_queue.cc:1035] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:42.047298 7537 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:42.047737 32360 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:42.054401 29096 catalog_manager.cc:5484] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:42.054886 29636 raft_consensus.cc:2932] T e5ed5a5797424f0f88c2b3c711468e6b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:42.069665 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed5a5797424f0f88c2b3c711468e6b with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:42.073410 7747 tablet_replica.cc:330] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:42.073621 7747 raft_consensus.cc:2229] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:42.073822 7747 raft_consensus.cc:2258] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:42.074824 7747 ts_tablet_manager.cc:1875] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:42.080319 7747 ts_tablet_manager.cc:1888] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:42.085103 7747 log.cc:1196] T e5ed5a5797424f0f88c2b3c711468e6b P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/e5ed5a5797424f0f88c2b3c711468e6b
I20240501 14:01:42.086064 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet e5ed5a5797424f0f88c2b3c711468e6b (table rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]) successfully deleted
I20240501 14:01:42.389593 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:42.397923 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:42.404516 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:42.442438 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:42.458418 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:42.465188 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:42.493435 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:42.532586 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:42.688892 29636 consensus_queue.cc:237] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:42.690860 31964 raft_consensus.cc:1261] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:42.690956 29245 raft_consensus.cc:1261] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:42.691397 7397 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:42.691665 6668 consensus_queue.cc:1035] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:42.694552 7397 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:42.698043 29245 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:42.711165 31964 raft_consensus.cc:2932] T ab3c1d804ba74db794bc0abe68bc9f96 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:42.711799 29102 catalog_manager.cc:5484] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:42.725221 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c1d804ba74db794bc0abe68bc9f96 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:42.729813 7833 tablet_replica.cc:330] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:42.730072 7833 raft_consensus.cc:2229] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:42.730296 7833 raft_consensus.cc:2258] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:42.730890 7833 ts_tablet_manager.cc:1875] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:42.735706 7833 ts_tablet_manager.cc:1888] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:42.735823 7833 log.cc:1196] T ab3c1d804ba74db794bc0abe68bc9f96 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/ab3c1d804ba74db794bc0abe68bc9f96
I20240501 14:01:42.736567 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet ab3c1d804ba74db794bc0abe68bc9f96 (table rebalancer_extra_table_42 [id=bfd9678bcb5846e7b9c770bb255fb453]) successfully deleted
I20240501 14:01:43.101167 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:43.105774 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:43.125481 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:43.138679 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:43.184181 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:43.226119 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:43.260694 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:43.279510 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:43.391036 29244 consensus_queue.cc:237] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:43.393132 32097 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:43.393229 29372 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:43.394014 7865 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:43.394346 6842 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:43.397811 7865 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:43.398414 32097 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:43.399235 29372 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:43.404279 29099 catalog_manager.cc:5484] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:43.419521 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet c6973301bdd94c2b8f98e718715ff752 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:43.420711 7922 tablet_replica.cc:330] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:43.424261 7922 raft_consensus.cc:2229] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:43.424574 7922 raft_consensus.cc:2258] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:43.425371 7922 ts_tablet_manager.cc:1875] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:43.432329 7922 ts_tablet_manager.cc:1888] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:43.432545 7922 log.cc:1196] T c6973301bdd94c2b8f98e718715ff752 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6973301bdd94c2b8f98e718715ff752
I20240501 14:01:43.433359 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet c6973301bdd94c2b8f98e718715ff752 (table rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]) successfully deleted
I20240501 14:01:43.707973 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:43.745956 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:43.774103 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:43.775084 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:43.818485 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:43.847728 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:43.883845 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:43.961766 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:44.111191 29244 consensus_queue.cc:237] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:44.113169 29372 raft_consensus.cc:1261] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:44.113289 29506 raft_consensus.cc:1261] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:44.113723 7865 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:44.114123 6842 consensus_queue.cc:1035] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:44.116706 7996 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:44.117430 29372 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:44.122821 29099 catalog_manager.cc:5484] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:44.131736 29506 raft_consensus.cc:2932] T e67dc79a086048a8839ae386857652ea P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:44.149456 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet e67dc79a086048a8839ae386857652ea with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:44.150125 8009 tablet_replica.cc:330] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:44.150418 8009 raft_consensus.cc:2229] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:44.150724 8009 raft_consensus.cc:2258] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:44.151453 8009 ts_tablet_manager.cc:1875] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:44.160934 8009 ts_tablet_manager.cc:1888] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:44.161152 8009 log.cc:1196] T e67dc79a086048a8839ae386857652ea P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/e67dc79a086048a8839ae386857652ea
I20240501 14:01:44.162101 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet e67dc79a086048a8839ae386857652ea (table rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]) successfully deleted
I20240501 14:01:44.441741 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:44.453397 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:44.504753 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:44.521026 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:44.524330 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:44.532846 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:44.590886 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:44.609961 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:44.759778 29637 consensus_queue.cc:237] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:44.762076 32096 raft_consensus.cc:1261] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:44.762100 29244 raft_consensus.cc:1261] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:44.763389 7397 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:44.764267 7397 consensus_queue.cc:1035] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:44.766058 6668 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:44.766968 32096 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:44.770325 29244 raft_consensus.cc:2932] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:44.773762 29097 catalog_manager.cc:5484] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:44.798748 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:44.801290 8087 tablet_replica.cc:330] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:44.801604 8087 raft_consensus.cc:2229] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:44.801862 8087 raft_consensus.cc:2258] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:44.802575 8087 ts_tablet_manager.cc:1875] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:44.809545 8087 ts_tablet_manager.cc:1888] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:44.809675 8087 log.cc:1196] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/7ec9bb1d59ac4b11a1f9b0d44b1dc69b
I20240501 14:01:44.810356 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b (table rebalancer_extra_table_14 [id=addcc12734134114bcea55a3c97f9d25]) successfully deleted
I20240501 14:01:45.202605 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:45.231002 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:45.272166 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:45.286099 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:45.301156 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:45.340898 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:45.371003 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:45.393088 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:45.554160 29506 consensus_queue.cc:237] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:45.556314 32359 raft_consensus.cc:1261] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.556494 29637 raft_consensus.cc:1261] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.557235 7893 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.557750 7893 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.561749 8169 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:45.565024 32359 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:45.576071 29637 raft_consensus.cc:2932] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:45.576665 29096 catalog_manager.cc:5484] T d2e5851626b54e1caf6105a70ccafc43 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:45.588696 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e5851626b54e1caf6105a70ccafc43 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51134
I20240501 14:01:45.591126 8180 tablet_replica.cc:330] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:01:45.591365 8180 raft_consensus.cc:2229] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.591585 8180 raft_consensus.cc:2258] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.592427 8180 ts_tablet_manager.cc:1875] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.599082 8180 ts_tablet_manager.cc:1888] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:45.599205 8180 log.cc:1196] T d2e5851626b54e1caf6105a70ccafc43 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/d2e5851626b54e1caf6105a70ccafc43
I20240501 14:01:45.600140 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet d2e5851626b54e1caf6105a70ccafc43 (table rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]) successfully deleted
I20240501 14:01:45.870482 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:45.877091 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:45.888619 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:45.977989 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:46.026388 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:46.030073 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:46.040239 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:46.053300 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:46.273478 32359 consensus_queue.cc:237] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:46.275887 29636 raft_consensus.cc:1261] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:46.275875 32097 raft_consensus.cc:1261] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:46.276587 6950 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:46.276888 7875 consensus_queue.cc:1035] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:46.281690 7875 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:46.282450 32097 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:46.286293 29636 raft_consensus.cc:2932] T 36b0222ebb6d4246ab8551d5579b1760 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:46.289693 29105 catalog_manager.cc:5484] T 36b0222ebb6d4246ab8551d5579b1760 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:46.306536 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b0222ebb6d4246ab8551d5579b1760 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:01:46.316757 8266 tablet_replica.cc:330] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:01:46.317047 8266 raft_consensus.cc:2229] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.317442 8266 raft_consensus.cc:2258] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.318228 8266 ts_tablet_manager.cc:1875] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.325115 8266 ts_tablet_manager.cc:1888] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:46.325269 8266 log.cc:1196] T 36b0222ebb6d4246ab8551d5579b1760 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/36b0222ebb6d4246ab8551d5579b1760
I20240501 14:01:46.326102 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet 36b0222ebb6d4246ab8551d5579b1760 (table rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]) successfully deleted
I20240501 14:01:46.708153 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:46.712033 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:46.741930 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:46.744302 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:46.759447 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:46.814371 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:46.847575 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:46.868777 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:47.016227 32223 consensus_queue.cc:237] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:47.019762 29372 raft_consensus.cc:1261] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:47.020027 31965 raft_consensus.cc:1261] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:47.020298 8144 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:47.020802 8144 consensus_queue.cc:1035] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:47.024538 8144 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.025315 29372 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.025220 31964 raft_consensus.cc:2932] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.031977 29097 catalog_manager.cc:5484] T 25a6e3129a2f4bf89a36cdca5eaeb376 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.037811 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a6e3129a2f4bf89a36cdca5eaeb376 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:47.038627 8359 tablet_replica.cc:330] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:47.038915 8359 raft_consensus.cc:2229] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:47.039295 8359 raft_consensus.cc:2258] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:47.039875 8359 ts_tablet_manager.cc:1875] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:47.045680 8359 ts_tablet_manager.cc:1888] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:47.045907 8359 log.cc:1196] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/25a6e3129a2f4bf89a36cdca5eaeb376
I20240501 14:01:47.047623 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 25a6e3129a2f4bf89a36cdca5eaeb376 (table rebalancer_extra_table_28 [id=c9fd45bb577d4470914ae91fa90bcaeb]) successfully deleted
I20240501 14:01:47.126986 32359 consensus_queue.cc:237] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:47.129696 32223 raft_consensus.cc:1261] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:47.129696 29372 raft_consensus.cc:1261] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:47.130295 6950 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:47.133738 32294 consensus_peers.cc:485] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 356722d5ec9d4d3d8ba08b4e7c2b2b11. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.135368 8258 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:47.140319 8401 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:47.140913 32223 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:47.147750 29101 catalog_manager.cc:5484] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:47.163458 29372 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:47.258330 29636 consensus_queue.cc:237] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:47.260879 29372 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:47.261355 29572 consensus_peers.cc:485] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 192a6cc308da4a448f03717b6a439094. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.261420 29506 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:47.261680 8149 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.262347 8149 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.266459 7397 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.270612 29506 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.273505 29372 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.276007 29101 catalog_manager.cc:5484] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.333911 31965 consensus_queue.cc:237] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:47.343109 32223 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:47.344095 32097 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:47.343983 8443 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:47.344722 7725 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:47.345073 31900 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa4a1cdf1c214e929b1d81ae3df585d9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.347573 7725 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.347967 32097 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.352166 32223 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.354876 29104 catalog_manager.cc:5484] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:47.394256 29372 consensus_queue.cc:237] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:47.398556 29636 raft_consensus.cc:1261] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:47.398905 29506 raft_consensus.cc:1261] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:47.399027 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87b1de51c4c4f909c27c27aa745fb44. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.399274 8187 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.399502 7891 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.411231 8187 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:47.411967 29636 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:47.415836 29506 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:47.419674 29097 catalog_manager.cc:5484] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:01:47.466655 29245 consensus_queue.cc:237] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:01:47.492626 29181 consensus_peers.cc:485] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d39eab213cc464ba90149276cfeeac8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.492609 29636 raft_consensus.cc:1261] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:47.493350 8335 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.497303 29506 raft_consensus.cc:1261] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:47.498065 8335 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.538378 8238 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.539330 29636 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.545964 29506 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.548272 29104 catalog_manager.cc:5484] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:47.701800 8512 ts_tablet_manager.cc:902] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:01:47.728001 8512 tablet_copy_client.cc:308] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:01:47.728679 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:49239)
I20240501 14:01:47.728853 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:49239: session id = 6518b7a04854458bb2fb23edc8b6f7ba-356722d5ec9d4d3d8ba08b4e7c2b2b11
I20240501 14:01:47.730969 32383 tablet_copy_source_session.cc:205] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.732352 8512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.742280 8512 tablet_copy_client.cc:783] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.742713 8512 tablet_copy_client.cc:647] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.746403 8512 tablet_copy_client.cc:523] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.759300 8512 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:47.771003 8512 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:47.771756 8512 tablet_bootstrap.cc:492] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:47.772181 8512 ts_tablet_manager.cc:1367] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20240501 14:01:47.773347 8512 raft_consensus.cc:348] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:47.773800 8512 raft_consensus.cc:729] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:47.774269 8512 consensus_queue.cc:260] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:47.779603 8512 ts_tablet_manager.cc:1398] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:47.780493 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-356722d5ec9d4d3d8ba08b4e7c2b2b11 received from {username='slave'} at 127.22.33.65:49239
I20240501 14:01:47.780634 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-356722d5ec9d4d3d8ba08b4e7c2b2b11 on tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:47.878919 8534 ts_tablet_manager.cc:902] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:01:47.893105 29571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 54460.7us
I20240501 14:01:47.893785 8533 ts_tablet_manager.cc:902] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:47.900902 8534 tablet_copy_client.cc:308] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:47.901749 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 192a6cc308da4a448f03717b6a439094 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:60765)
I20240501 14:01:47.901921 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 192a6cc308da4a448f03717b6a439094 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:60765: session id = 9903ed35bca144bcb74e9bf5102117e7-192a6cc308da4a448f03717b6a439094
I20240501 14:01:47.904227 29657 tablet_copy_source_session.cc:205] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.907281 8534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192a6cc308da4a448f03717b6a439094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.950596 8549 ts_tablet_manager.cc:902] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:47.959951 8543 ts_tablet_manager.cc:902] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:47.960471 8533 tablet_copy_client.cc:308] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:47.961418 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 9d39eab213cc464ba90149276cfeeac8 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:44499)
I20240501 14:01:47.961606 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 9d39eab213cc464ba90149276cfeeac8 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:44499: session id = 9903ed35bca144bcb74e9bf5102117e7-9d39eab213cc464ba90149276cfeeac8
I20240501 14:01:47.964785 29266 tablet_copy_source_session.cc:205] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.971050 8533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d39eab213cc464ba90149276cfeeac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.982450 8533 tablet_copy_client.cc:783] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.986265 8533 tablet_copy_client.cc:647] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:48.001837 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:48.006677 8533 tablet_copy_client.cc:523] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.013703 8533 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:48.018805 8534 tablet_copy_client.cc:783] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:48.019074 8534 tablet_copy_client.cc:647] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:48.054174 8543 tablet_copy_client.cc:308] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:01:48.056038 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet fa4a1cdf1c214e929b1d81ae3df585d9 from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:57827)
I20240501 14:01:48.056234 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet fa4a1cdf1c214e929b1d81ae3df585d9 from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:57827: session id = b87d188552494b4590afc37a85b98c35-fa4a1cdf1c214e929b1d81ae3df585d9
I20240501 14:01:48.057399 8534 tablet_copy_client.cc:523] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.059253 31985 tablet_copy_source_session.cc:205] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:48.068308 8543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4a1cdf1c214e929b1d81ae3df585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:48.076403 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:48.082851 8543 tablet_copy_client.cc:783] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:48.083102 8543 tablet_copy_client.cc:647] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:48.084663 8543 tablet_copy_client.cc:523] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.088443 8543 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:48.098760 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:48.106799 29246 raft_consensus.cc:1203] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:48.122603 8549 tablet_copy_client.cc:308] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:48.123452 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet a87b1de51c4c4f909c27c27aa745fb44 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:57801)
I20240501 14:01:48.123651 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet a87b1de51c4c4f909c27c27aa745fb44 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:57801: session id = 6a446ac802a7457d9f27076781f21d79-a87b1de51c4c4f909c27c27aa745fb44
I20240501 14:01:48.127463 29397 tablet_copy_source_session.cc:205] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:48.150808 8549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87b1de51c4c4f909c27c27aa745fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:48.182896 8534 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:48.191006 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:48.196503 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:48.243906 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:01:48.280849 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 47755.4us
I20240501 14:01:48.304025 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:48.305609 8549 tablet_copy_client.cc:783] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:48.305785 8533 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.305997 8549 tablet_copy_client.cc:647] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:48.306219 8533 tablet_bootstrap.cc:492] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:01:48.306488 8533 ts_tablet_manager.cc:1367] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.293s user 0.084s sys 0.063s
I20240501 14:01:48.307327 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:48.307301 8533 raft_consensus.cc:348] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:48.307536 8533 raft_consensus.cc:729] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:48.307756 8533 consensus_queue.cc:260] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:48.317363 8533 ts_tablet_manager.cc:1398] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240501 14:01:48.318579 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-9d39eab213cc464ba90149276cfeeac8 received from {username='slave'} at 127.22.33.69:44499
I20240501 14:01:48.318702 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-9d39eab213cc464ba90149276cfeeac8 on tablet 9d39eab213cc464ba90149276cfeeac8 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:48.342686 8543 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.343233 8543 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:48.343493 8543 ts_tablet_manager.cc:1367] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.255s user 0.035s sys 0.087s
I20240501 14:01:48.344159 8543 raft_consensus.cc:348] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:48.344431 8543 raft_consensus.cc:729] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:48.344647 8543 consensus_queue.cc:260] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:48.345206 8543 ts_tablet_manager.cc:1398] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:48.347009 8549 tablet_copy_client.cc:523] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.347369 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-fa4a1cdf1c214e929b1d81ae3df585d9 received from {username='slave'} at 127.22.33.66:57827
I20240501 14:01:48.347488 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session b87d188552494b4590afc37a85b98c35-fa4a1cdf1c214e929b1d81ae3df585d9 on tablet fa4a1cdf1c214e929b1d81ae3df585d9 with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:48.366355 8549 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:01:48.403288 8534 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.403892 8534 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:01:48.404228 8534 ts_tablet_manager.cc:1367] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.343s user 0.048s sys 0.000s
I20240501 14:01:48.405104 8534 raft_consensus.cc:348] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:48.405418 8534 raft_consensus.cc:729] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:48.406461 8534 consensus_queue.cc:260] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:48.411561 8534 ts_tablet_manager.cc:1398] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:48.413770 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-192a6cc308da4a448f03717b6a439094 received from {username='slave'} at 127.22.33.69:60765
I20240501 14:01:48.413919 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-192a6cc308da4a448f03717b6a439094 on tablet 192a6cc308da4a448f03717b6a439094 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:48.441568 31964 raft_consensus.cc:1203] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.453567 31964 raft_consensus.cc:1203] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.509292 8549 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.509784 8549 tablet_bootstrap.cc:492] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:01:48.510057 8549 ts_tablet_manager.cc:1367] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.144s user 0.048s sys 0.000s
I20240501 14:01:48.510932 8549 raft_consensus.cc:348] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:48.511155 8549 raft_consensus.cc:729] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:01:48.511421 8549 consensus_queue.cc:260] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:01:48.513401 8549 ts_tablet_manager.cc:1398] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:48.515254 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-a87b1de51c4c4f909c27c27aa745fb44 received from {username='slave'} at 127.22.33.71:57801
I20240501 14:01:48.515381 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-a87b1de51c4c4f909c27c27aa745fb44 on tablet a87b1de51c4c4f909c27c27aa745fb44 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:01:48.873281 29372 raft_consensus.cc:1203] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:48.877240 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:48.884266 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:48.886096 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:48.909981 32224 raft_consensus.cc:1203] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.943959 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:48.991230 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:48.992117 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:48.995234 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:49.025851 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:49.241590 31964 consensus_queue.cc:237] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:49.243533 32222 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:49.243604 29372 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:49.244024 8607 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:49.244230 7342 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:49.247465 7342 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:49.248049 29372 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:49.255803 29098 catalog_manager.cc:5484] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:49.261380 32222 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:49.268229 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4a1cdf1c214e929b1d81ae3df585d9 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:01:49.277427 8669 tablet_replica.cc:330] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:01:49.277791 8669 raft_consensus.cc:2229] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.278050 8669 raft_consensus.cc:2258] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.278710 8669 ts_tablet_manager.cc:1875] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.285892 8669 ts_tablet_manager.cc:1888] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:49.286088 8669 log.cc:1196] T fa4a1cdf1c214e929b1d81ae3df585d9 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/fa4a1cdf1c214e929b1d81ae3df585d9
I20240501 14:01:49.286893 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet fa4a1cdf1c214e929b1d81ae3df585d9 (table rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]) successfully deleted
I20240501 14:01:49.626140 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:49.629222 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:49.641261 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:49.653091 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:49.678158 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:49.686578 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:49.687618 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:49.731984 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:49.890697 29245 consensus_queue.cc:237] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:49.892536 29636 raft_consensus.cc:1261] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.892588 31964 raft_consensus.cc:1261] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.893141 8576 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.893436 8238 consensus_queue.cc:1035] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.896602 8238 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:49.897665 31964 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:49.900748 29636 raft_consensus.cc:2932] T 9d39eab213cc464ba90149276cfeeac8 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:49.906919 29098 catalog_manager.cc:5484] T 9d39eab213cc464ba90149276cfeeac8 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:49.929852 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d39eab213cc464ba90149276cfeeac8 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:49.941458 8753 tablet_replica.cc:330] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:49.941735 8753 raft_consensus.cc:2229] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.941977 8753 raft_consensus.cc:2258] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.942775 8753 ts_tablet_manager.cc:1875] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.948405 8753 ts_tablet_manager.cc:1888] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:49.948581 8753 log.cc:1196] T 9d39eab213cc464ba90149276cfeeac8 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d39eab213cc464ba90149276cfeeac8
I20240501 14:01:49.949426 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 9d39eab213cc464ba90149276cfeeac8 (table rebalance_test_table_3 [id=90e554b79da745d1ad57288ed7f66c03]) successfully deleted
I20240501 14:01:50.219008 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:50.250800 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:50.265628 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:50.293977 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:50.315627 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:50.334538 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:50.359772 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:50.375825 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:50.590615 29636 consensus_queue.cc:237] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:50.592717 29372 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:50.592751 31964 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:50.593327 8814 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:50.593623 8532 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:50.596614 8814 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:50.597608 29372 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:50.599778 31964 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:50.603672 29101 catalog_manager.cc:5484] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:50.610545 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 192a6cc308da4a448f03717b6a439094 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:50.611922 8838 tablet_replica.cc:330] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:50.612144 8838 raft_consensus.cc:2229] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:50.612344 8838 raft_consensus.cc:2258] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:50.613195 8838 ts_tablet_manager.cc:1875] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:50.618214 8838 ts_tablet_manager.cc:1888] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:50.618345 8838 log.cc:1196] T 192a6cc308da4a448f03717b6a439094 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/192a6cc308da4a448f03717b6a439094
I20240501 14:01:50.619261 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 192a6cc308da4a448f03717b6a439094 (table rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]) successfully deleted
I20240501 14:01:50.892195 31944 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:01:50.935194 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:50.939777 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:50.948650 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:50.954313 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:50.965600 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:50.995862 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:50.997673 32342 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:51.134815 32359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08"
dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34632
I20240501 14:01:51.135069 32359 raft_consensus.cc:595] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:51.181493 8908 raft_consensus.cc:979] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: : Instructing follower 9903ed35bca144bcb74e9bf5102117e7 to start an election
I20240501 14:01:51.182016 8908 raft_consensus.cc:1067] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Signalling peer 9903ed35bca144bcb74e9bf5102117e7 to start an election
I20240501 14:01:51.183020 31964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08"
dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
from {username='slave'} at 127.22.33.72:34297
I20240501 14:01:51.183239 31964 raft_consensus.cc:482] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:51.183353 31964 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:51.185504 31964 raft_consensus.cc:504] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:51.186841 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:01:51.187057 32222 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:51.188997 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:51.189180 29636 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:51.189467 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aa71ecef4d419eb36c7777790e6f08" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:01:51.189636 32359 raft_consensus.cc:3032] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:51.189689 32359 raft_consensus.cc:729] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:01:51.189983 32359 consensus_queue.cc:260] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:51.190466 32359 raft_consensus.cc:3037] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:51.191958 29636 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:01:51.195653 31964 leader_election.cc:288] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:51.200491 32222 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:01:51.201359 32359 raft_consensus.cc:2454] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:01:51.201490 31899 leader_election.cc:302] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:51.201967 8607 raft_consensus.cc:2781] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:51.202100 8607 raft_consensus.cc:686] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:01:51.202392 8607 consensus_queue.cc:237] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:51.211480 29101 catalog_manager.cc:5484] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 1 to 2, leader changed from 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:51.477656 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:51.520465 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:51.531482 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:51.587026 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:51.616115 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:51.624939 32359 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:51.626104 8607 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:51.691701 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:51.693040 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:51.730557 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:51.736999 29635 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:51.738176 8607 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:51.763281 32224 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:51.765609 8977 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:52.339867 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:52.425479 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:52.462247 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:52.462880 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:52.480921 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:52.510303 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:52.511574 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:52.600345 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:52.723796 32359 consensus_queue.cc:237] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:52.725741 29377 raft_consensus.cc:1261] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:52.725904 29245 raft_consensus.cc:1261] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:52.726270 8908 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:52.726567 6950 consensus_queue.cc:1035] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:52.729097 6950 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.730549 29245 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.734956 29103 catalog_manager.cc:5484] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.735528 29377 raft_consensus.cc:2932] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.743975 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:52.744573 9118 tablet_replica.cc:330] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:52.744858 9118 raft_consensus.cc:2229] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:52.745097 9118 raft_consensus.cc:2258] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:52.745642 9118 ts_tablet_manager.cc:1875] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:52.753813 9118 ts_tablet_manager.cc:1888] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:52.753934 9118 log.cc:1196] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/356722d5ec9d4d3d8ba08b4e7c2b2b11
I20240501 14:01:52.754616 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11 (table rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]) successfully deleted
I20240501 14:01:52.854604 29245 consensus_queue.cc:237] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:52.862470 32222 raft_consensus.cc:1261] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:52.862722 32096 raft_consensus.cc:1261] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:52.863229 8746 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:52.864158 29180 consensus_peers.cc:485] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a025068d40c4b388421c5e6930847c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:52.864579 9094 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:52.881793 9094 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:52.882393 32224 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:52.887454 32096 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:52.890108 29105 catalog_manager.cc:5484] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:01:52.973243 32096 consensus_queue.cc:237] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:52.976383 29377 raft_consensus.cc:1261] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:52.976622 32030 consensus_peers.cc:485] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eb574dc421745618ba2a11f0b3f9110. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:52.977015 31965 raft_consensus.cc:1261] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:52.977778 7978 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:52.978332 7978 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:52.986490 9172 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.987222 31965 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.991828 29377 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:52.993767 29105 catalog_manager.cc:5484] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:53.048036 29506 consensus_queue.cc:237] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:01:53.051004 29442 consensus_peers.cc:485] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77ce342cbfad4419b3e2e8d3a52c4628. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.051193 29377 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.052438 29636 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.052958 8813 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.053401 9192 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.057060 8813 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.058318 29377 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.062198 29636 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.068780 29105 catalog_manager.cc:5484] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.102444 32096 consensus_queue.cc:237] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:53.105005 32359 raft_consensus.cc:1261] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.105093 32222 raft_consensus.cc:1261] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:53.105158 32030 consensus_peers.cc:485] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fce4b1a15bc48e9b8042869d5c18d16. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.105496 9172 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.105774 7978 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.108271 9172 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:53.108830 32222 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:53.115106 29102 catalog_manager.cc:5484] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:53.117700 32359 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:53.191475 29636 consensus_queue.cc:237] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:53.209080 29506 raft_consensus.cc:1261] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:53.209203 29572 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e0fcc86017b4589b1a960c078e7ec57. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.209968 9150 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.210258 32359 raft_consensus.cc:1261] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.210964 9150 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.226434 8425 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.231582 29506 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.233330 29103 catalog_manager.cc:5484] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.235416 32359 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.238469 9233 ts_tablet_manager.cc:902] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:01:53.249701 9233 tablet_copy_client.cc:308] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:01:53.250767 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 1a025068d40c4b388421c5e6930847c2 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:50987)
I20240501 14:01:53.250947 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 1a025068d40c4b388421c5e6930847c2 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:50987: session id = 6edd6a3430d44c38a19c2804e555059f-1a025068d40c4b388421c5e6930847c2
I20240501 14:01:53.253098 29266 tablet_copy_source_session.cc:205] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.254513 9233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a025068d40c4b388421c5e6930847c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.260409 9233 tablet_copy_client.cc:783] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.260625 9233 tablet_copy_client.cc:647] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.262676 9233 tablet_copy_client.cc:523] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.265944 9233 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:01:53.271422 9233 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.271754 9233 tablet_bootstrap.cc:492] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:01:53.271978 9233 ts_tablet_manager.cc:1367] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:53.272650 9233 raft_consensus.cc:348] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:53.272891 9233 raft_consensus.cc:729] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.273151 9233 consensus_queue.cc:260] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:01:53.273766 9233 ts_tablet_manager.cc:1398] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:53.276791 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-1a025068d40c4b388421c5e6930847c2 received from {username='slave'} at 127.22.33.72:50987
I20240501 14:01:53.276911 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-1a025068d40c4b388421c5e6930847c2 on tablet 1a025068d40c4b388421c5e6930847c2 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:01:53.295583 32097 consensus_queue.cc:237] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
W20240501 14:01:53.298102 32032 consensus_peers.cc:485] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815ce94772f2454ea755d33dd04a87d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.298175 32359 raft_consensus.cc:1261] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.298202 32222 raft_consensus.cc:1261] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.298686 8818 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.298944 7978 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.302073 7978 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:53.302681 32222 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:53.309195 32359 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:53.308908 29103 catalog_manager.cc:5484] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:01:53.358752 9273 ts_tablet_manager.cc:902] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:53.360098 9273 tablet_copy_client.cc:308] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:53.360762 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 1eb574dc421745618ba2a11f0b3f9110 from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:54889)
I20240501 14:01:53.360916 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 1eb574dc421745618ba2a11f0b3f9110 from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:54889: session id = 6518b7a04854458bb2fb23edc8b6f7ba-1eb574dc421745618ba2a11f0b3f9110
I20240501 14:01:53.363759 32117 tablet_copy_source_session.cc:205] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.364415 29372 consensus_queue.cc:237] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:53.366211 9273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb574dc421745618ba2a11f0b3f9110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.368001 32222 raft_consensus.cc:1261] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:53.369678 29310 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef4b00f102874e17830f2e340255a11f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.369829 8620 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.370005 29506 raft_consensus.cc:1261] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.370529 8620 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.374183 9273 tablet_copy_client.cc:783] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.373899 8983 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:53.374436 9273 tablet_copy_client.cc:647] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.374890 32222 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:53.375999 9273 tablet_copy_client.cc:523] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.381350 29506 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:53.385027 29102 catalog_manager.cc:5484] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:53.386453 9273 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:53.396402 9273 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.396800 9273 tablet_bootstrap.cc:492] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:53.397135 9273 ts_tablet_manager.cc:1367] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:01:53.397809 9273 raft_consensus.cc:348] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:53.398013 9273 raft_consensus.cc:729] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.398258 9273 consensus_queue.cc:260] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:53.402359 9273 ts_tablet_manager.cc:1398] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:53.403095 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-1eb574dc421745618ba2a11f0b3f9110 received from {username='slave'} at 127.22.33.65:54889
I20240501 14:01:53.403250 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-1eb574dc421745618ba2a11f0b3f9110 on tablet 1eb574dc421745618ba2a11f0b3f9110 with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:53.438798 29635 consensus_queue.cc:237] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:53.441051 29506 raft_consensus.cc:1261] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.441128 29245 raft_consensus.cc:1261] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.441713 9150 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.441946 8532 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:53.442498 29572 consensus_peers.cc:485] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ce8f6f27b4ab19ff5a1bb972ca202. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.446445 8425 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.449651 29246 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.458385 29506 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.461782 29105 catalog_manager.cc:5484] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:53.541985 29635 consensus_queue.cc:237] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:53.546018 29506 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.546761 8425 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:53.547053 29572 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f345c7a43d2648b4b4c47e726c4d4c51. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.547705 29372 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.548269 8532 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.565753 9316 ts_tablet_manager.cc:902] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:53.565899 8425 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.566583 29507 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.571924 29372 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.574061 29105 catalog_manager.cc:5484] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.597998 9316 tablet_copy_client.cc:308] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:53.598698 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 77ce342cbfad4419b3e2e8d3a52c4628 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:54045)
I20240501 14:01:53.598856 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 77ce342cbfad4419b3e2e8d3a52c4628 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:54045: session id = 9903ed35bca144bcb74e9bf5102117e7-77ce342cbfad4419b3e2e8d3a52c4628
I20240501 14:01:53.601166 29527 tablet_copy_source_session.cc:205] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.602648 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce342cbfad4419b3e2e8d3a52c4628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.608275 9316 tablet_copy_client.cc:783] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.608530 9316 tablet_copy_client.cc:647] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.610198 9316 tablet_copy_client.cc:523] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.614269 9316 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:53.653295 32097 consensus_queue.cc:237] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:53.656327 32359 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:53.656383 9316 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.656790 9316 tablet_bootstrap.cc:492] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:01:53.657109 9316 ts_tablet_manager.cc:1367] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.007s
I20240501 14:01:53.657128 7978 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.657421 32223 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:53.657687 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54bdc21d1dc644e7ac44f5b9ad39ab32. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.657914 9316 raft_consensus.cc:348] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:53.658159 9316 raft_consensus.cc:729] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.658401 9316 consensus_queue.cc:260] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:53.658452 8818 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:53.659695 9344 ts_tablet_manager.cc:902] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:53.660368 9316 ts_tablet_manager.cc:1398] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:53.661010 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-77ce342cbfad4419b3e2e8d3a52c4628 received from {username='slave'} at 127.22.33.69:54045
I20240501 14:01:53.661130 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-77ce342cbfad4419b3e2e8d3a52c4628 on tablet 77ce342cbfad4419b3e2e8d3a52c4628 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:53.674751 9344 tablet_copy_client.cc:308] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:53.674959 7978 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:53.675381 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 7fce4b1a15bc48e9b8042869d5c18d16 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57709)
I20240501 14:01:53.675529 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 7fce4b1a15bc48e9b8042869d5c18d16 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57709: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-7fce4b1a15bc48e9b8042869d5c18d16
I20240501 14:01:53.675616 32359 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:53.678261 32117 tablet_copy_source_session.cc:205] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.679713 9344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fce4b1a15bc48e9b8042869d5c18d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.681735 29103 catalog_manager.cc:5484] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:53.687844 9344 tablet_copy_client.cc:783] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.688078 9344 tablet_copy_client.cc:647] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.687908 32223 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:01:53.695082 9344 tablet_copy_client.cc:523] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.699244 9344 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:01:53.708662 9344 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.709049 9344 tablet_bootstrap.cc:492] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:01:53.709324 9344 ts_tablet_manager.cc:1367] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240501 14:01:53.710018 9344 raft_consensus.cc:348] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:53.710259 9344 raft_consensus.cc:729] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.710847 9344 consensus_queue.cc:260] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:53.712769 9344 ts_tablet_manager.cc:1398] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:53.713450 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-7fce4b1a15bc48e9b8042869d5c18d16 received from {username='slave'} at 127.22.33.68:57709
I20240501 14:01:53.713557 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-7fce4b1a15bc48e9b8042869d5c18d16 on tablet 7fce4b1a15bc48e9b8042869d5c18d16 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:53.739102 29507 consensus_queue.cc:237] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
W20240501 14:01:53.742208 29442 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21c02994dfef4a10a87b1f47d3ab6e3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:53.742251 29636 raft_consensus.cc:1261] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.742511 29377 raft_consensus.cc:1261] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:53.742882 9192 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.743157 8813 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:53.746699 9192 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.747156 29377 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.751444 29636 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.754367 29102 catalog_manager.cc:5484] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:53.766119 32359 raft_consensus.cc:1203] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:53.768473 9376 ts_tablet_manager.cc:902] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:53.779261 9376 tablet_copy_client.cc:308] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:53.779932 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 815ce94772f2454ea755d33dd04a87d2 from peer b87d188552494b4590afc37a85b98c35 ({username='slave'} at 127.22.33.66:42885)
I20240501 14:01:53.780108 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 815ce94772f2454ea755d33dd04a87d2 from peer b87d188552494b4590afc37a85b98c35 at {username='slave'} at 127.22.33.66:42885: session id = b87d188552494b4590afc37a85b98c35-815ce94772f2454ea755d33dd04a87d2
I20240501 14:01:53.783434 32117 tablet_copy_source_session.cc:205] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.785151 9376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815ce94772f2454ea755d33dd04a87d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.791558 9376 tablet_copy_client.cc:783] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.791841 9376 tablet_copy_client.cc:647] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.793653 9376 tablet_copy_client.cc:523] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.800462 9376 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Bootstrap starting.
I20240501 14:01:53.815538 9273 ts_tablet_manager.cc:902] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:53.815912 9387 ts_tablet_manager.cc:902] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:53.842105 9376 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.842593 9376 tablet_bootstrap.cc:492] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Bootstrap complete.
I20240501 14:01:53.842857 9376 ts_tablet_manager.cc:1367] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.022s
I20240501 14:01:53.844468 9316 ts_tablet_manager.cc:902] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:53.855085 9376 raft_consensus.cc:348] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:53.855369 9376 raft_consensus.cc:729] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d188552494b4590afc37a85b98c35, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.855602 9376 consensus_queue.cc:260] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:01:53.859146 9376 ts_tablet_manager.cc:1398] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35: Time spent starting tablet: real 0.016s user 0.000s sys 0.004s
I20240501 14:01:53.860884 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session b87d188552494b4590afc37a85b98c35-815ce94772f2454ea755d33dd04a87d2 received from {username='slave'} at 127.22.33.66:42885
I20240501 14:01:53.861018 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session b87d188552494b4590afc37a85b98c35-815ce94772f2454ea755d33dd04a87d2 on tablet 815ce94772f2454ea755d33dd04a87d2 with peer b87d188552494b4590afc37a85b98c35
I20240501 14:01:53.871661 9387 tablet_copy_client.cc:308] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:53.872547 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 7e0fcc86017b4589b1a960c078e7ec57 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:48851)
I20240501 14:01:53.872704 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 7e0fcc86017b4589b1a960c078e7ec57 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:48851: session id = 0a697f3a1ee544daa7073814ab845d58-7e0fcc86017b4589b1a960c078e7ec57
I20240501 14:01:53.876475 29657 tablet_copy_source_session.cc:205] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.878049 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0fcc86017b4589b1a960c078e7ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.880352 9273 tablet_copy_client.cc:308] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:01:53.881330 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet ef4b00f102874e17830f2e340255a11f from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:35753)
I20240501 14:01:53.881569 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet ef4b00f102874e17830f2e340255a11f from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:35753: session id = 6518b7a04854458bb2fb23edc8b6f7ba-ef4b00f102874e17830f2e340255a11f
I20240501 14:01:53.890897 29397 tablet_copy_source_session.cc:205] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.891845 9387 tablet_copy_client.cc:783] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.893505 9387 tablet_copy_client.cc:647] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.895340 9387 tablet_copy_client.cc:523] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.901662 9273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b00f102874e17830f2e340255a11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.903205 9387 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:53.916781 9273 tablet_copy_client.cc:783] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.917136 9273 tablet_copy_client.cc:647] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.919359 9273 tablet_copy_client.cc:523] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.925737 9316 tablet_copy_client.cc:308] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:53.926505 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:60765)
I20240501 14:01:53.926673 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:60765: session id = 9903ed35bca144bcb74e9bf5102117e7-3d8ce8f6f27b4ab19ff5a1bb972ca202
I20240501 14:01:53.930269 29246 raft_consensus.cc:1203] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:53.934271 29657 tablet_copy_source_session.cc:205] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:53.937342 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:53.940541 9387 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.941115 9387 tablet_bootstrap.cc:492] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:53.941442 9387 ts_tablet_manager.cc:1367] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.007s
I20240501 14:01:53.942327 9387 raft_consensus.cc:348] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:53.942636 9387 raft_consensus.cc:729] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.938351 9273 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:01:53.943729 9387 consensus_queue.cc:260] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:53.945731 9387 ts_tablet_manager.cc:1398] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:01:53.948818 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-7e0fcc86017b4589b1a960c078e7ec57 received from {username='slave'} at 127.22.33.70:48851
I20240501 14:01:53.948941 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-7e0fcc86017b4589b1a960c078e7ec57 on tablet 7e0fcc86017b4589b1a960c078e7ec57 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:53.956585 9273 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.957238 9273 tablet_bootstrap.cc:492] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:01:53.957480 9273 ts_tablet_manager.cc:1367] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20240501 14:01:53.958138 9273 raft_consensus.cc:348] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:53.958344 9273 raft_consensus.cc:729] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:01:53.958527 9316 tablet_copy_client.cc:783] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:53.958777 9316 tablet_copy_client.cc:647] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:53.958909 9273 consensus_queue.cc:260] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:53.960968 9316 tablet_copy_client.cc:523] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.961390 9273 ts_tablet_manager.cc:1398] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:01:53.962100 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-ef4b00f102874e17830f2e340255a11f received from {username='slave'} at 127.22.33.65:35753
I20240501 14:01:53.962218 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-ef4b00f102874e17830f2e340255a11f on tablet ef4b00f102874e17830f2e340255a11f with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:01:53.964855 9316 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:01:54.021880 31965 raft_consensus.cc:1203] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:54.022745 9316 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.023111 9316 tablet_bootstrap.cc:492] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:01:54.023391 9316 ts_tablet_manager.cc:1367] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.059s user 0.048s sys 0.004s
I20240501 14:01:54.024220 9316 raft_consensus.cc:348] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:54.024518 9316 raft_consensus.cc:729] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:01:54.024776 9316 consensus_queue.cc:260] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:54.025666 9316 ts_tablet_manager.cc:1398] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:54.026844 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-3d8ce8f6f27b4ab19ff5a1bb972ca202 received from {username='slave'} at 127.22.33.69:60765
I20240501 14:01:54.026955 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-3d8ce8f6f27b4ab19ff5a1bb972ca202 on tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:01:54.113339 9430 ts_tablet_manager.cc:902] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:01:54.127570 9430 tablet_copy_client.cc:308] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:01:54.128696 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:47475)
I20240501 14:01:54.128863 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:47475: session id = 522f4ff5e9084cbf8462196533c7d7aa-54bdc21d1dc644e7ac44f5b9ad39ab32
I20240501 14:01:54.131227 32117 tablet_copy_source_session.cc:205] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:54.138010 9430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:54.144464 9430 tablet_copy_client.cc:783] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:54.144793 9430 tablet_copy_client.cc:647] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:54.148542 9430 tablet_copy_client.cc:523] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.151474 9387 ts_tablet_manager.cc:902] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:01:54.161608 9430 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:01:54.183681 9430 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.184494 9430 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:01:54.184767 9430 ts_tablet_manager.cc:1367] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.002s
I20240501 14:01:54.185585 9430 raft_consensus.cc:348] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:54.185904 9430 raft_consensus.cc:729] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:01:54.186244 9430 consensus_queue.cc:260] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:01:54.188586 9430 ts_tablet_manager.cc:1398] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:54.193893 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-54bdc21d1dc644e7ac44f5b9ad39ab32 received from {username='slave'} at 127.22.33.67:47475
I20240501 14:01:54.194090 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-54bdc21d1dc644e7ac44f5b9ad39ab32 on tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:01:54.201855 29635 raft_consensus.cc:1203] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:54.227909 9387 tablet_copy_client.cc:308] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:01:54.228837 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet f345c7a43d2648b4b4c47e726c4d4c51 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:48851)
I20240501 14:01:54.229002 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet f345c7a43d2648b4b4c47e726c4d4c51 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:48851: session id = 0a697f3a1ee544daa7073814ab845d58-f345c7a43d2648b4b4c47e726c4d4c51
I20240501 14:01:54.232863 29657 tablet_copy_source_session.cc:205] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:54.240362 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345c7a43d2648b4b4c47e726c4d4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:54.247870 9387 tablet_copy_client.cc:783] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:54.248163 9387 tablet_copy_client.cc:647] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:54.250100 9387 tablet_copy_client.cc:523] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.258112 9387 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:54.270730 9450 ts_tablet_manager.cc:902] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:01:54.274807 29246 raft_consensus.cc:1203] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:54.329710 9450 tablet_copy_client.cc:308] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:01:54.331723 29527 tablet_copy_service.cc:140] P 522f4ff5e9084cbf8462196533c7d7aa: Received BeginTabletCopySession request for tablet 21c02994dfef4a10a87b1f47d3ab6e3b from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36525)
I20240501 14:01:54.331877 29527 tablet_copy_service.cc:161] P 522f4ff5e9084cbf8462196533c7d7aa: Beginning new tablet copy session on tablet 21c02994dfef4a10a87b1f47d3ab6e3b from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36525: session id = 0a697f3a1ee544daa7073814ab845d58-21c02994dfef4a10a87b1f47d3ab6e3b
I20240501 14:01:54.334187 29527 tablet_copy_source_session.cc:205] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:54.336601 29372 raft_consensus.cc:1203] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:54.347478 9450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c02994dfef4a10a87b1f47d3ab6e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:54.348655 9387 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.350833 9387 tablet_bootstrap.cc:492] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:54.351181 9387 ts_tablet_manager.cc:1367] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.094s user 0.045s sys 0.003s
I20240501 14:01:54.362711 32097 raft_consensus.cc:1203] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:54.363257 9387 raft_consensus.cc:348] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:54.363520 9387 raft_consensus.cc:729] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:54.363930 9387 consensus_queue.cc:260] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:54.371444 9450 tablet_copy_client.cc:783] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:54.371727 9450 tablet_copy_client.cc:647] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:54.376571 9450 tablet_copy_client.cc:523] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.383635 9450 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:01:54.391865 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:54.401504 9387 ts_tablet_manager.cc:1398] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.050s user 0.008s sys 0.004s
I20240501 14:01:54.404062 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-f345c7a43d2648b4b4c47e726c4d4c51 received from {username='slave'} at 127.22.33.70:48851
I20240501 14:01:54.404220 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-f345c7a43d2648b4b4c47e726c4d4c51 on tablet f345c7a43d2648b4b4c47e726c4d4c51 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:54.405534 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:54.450742 31964 raft_consensus.cc:1203] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:54.473832 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:54.524240 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:54.528939 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:54.549052 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:54.555778 9450 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.573751 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:54.592698 29507 raft_consensus.cc:1203] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:54.606351 9450 tablet_bootstrap.cc:492] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:01:54.606813 9450 ts_tablet_manager.cc:1367] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.223s user 0.081s sys 0.004s
I20240501 14:01:54.607887 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:54.607681 9450 raft_consensus.cc:348] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:54.607980 9450 raft_consensus.cc:729] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:01:54.608366 9450 consensus_queue.cc:260] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:54.616885 9450 ts_tablet_manager.cc:1398] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:01:54.617798 29527 tablet_copy_service.cc:342] P 522f4ff5e9084cbf8462196533c7d7aa: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-21c02994dfef4a10a87b1f47d3ab6e3b received from {username='slave'} at 127.22.33.70:36525
I20240501 14:01:54.617913 29527 tablet_copy_service.cc:433] P 522f4ff5e9084cbf8462196533c7d7aa: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-21c02994dfef4a10a87b1f47d3ab6e3b on tablet 21c02994dfef4a10a87b1f47d3ab6e3b with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:01:54.657886 32096 raft_consensus.cc:1203] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:54.763510 32096 raft_consensus.cc:1203] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:55.067080 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:55.069916 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:55.070531 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:55.094520 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:55.130704 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:55.142339 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:55.186137 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:55.268711 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:55.385965 29635 consensus_queue.cc:237] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:55.388218 31965 raft_consensus.cc:1261] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:55.388231 29246 raft_consensus.cc:1261] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:55.388741 9150 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:55.389005 8425 consensus_queue.cc:1035] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:55.392036 9150 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:55.395532 31964 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:55.404112 29096 catalog_manager.cc:5484] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:55.413134 29245 raft_consensus.cc:2932] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:55.423785 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:01:55.424423 9581 tablet_replica.cc:330] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:01:55.424651 9581 raft_consensus.cc:2229] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.424921 9581 raft_consensus.cc:2258] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.426163 9581 ts_tablet_manager.cc:1875] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.433179 9581 ts_tablet_manager.cc:1888] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:01:55.433537 9581 log.cc:1196] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d8ce8f6f27b4ab19ff5a1bb972ca202
I20240501 14:01:55.434520 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202 (table rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]) successfully deleted
I20240501 14:01:55.708220 32076 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:01:55.762359 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:55.774320 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:55.776582 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:55.790745 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:55.841727 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:55.849599 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:55.875308 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:01:55.927629 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 70949.8us
W20240501 14:01:55.929627 32296 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 79357.6us
I20240501 14:01:56.025085 29635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51"
dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56286
I20240501 14:01:56.025331 29635 raft_consensus.cc:595] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:56.203742 9563 raft_consensus.cc:979] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: : Instructing follower 0a697f3a1ee544daa7073814ab845d58 to start an election
I20240501 14:01:56.203878 9563 raft_consensus.cc:1067] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Signalling peer 0a697f3a1ee544daa7073814ab845d58 to start an election
I20240501 14:01:56.205674 32096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51"
dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
from {username='slave'} at 127.22.33.68:57709
I20240501 14:01:56.205936 32096 raft_consensus.cc:482] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:56.206043 32096 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:56.215720 32096 raft_consensus.cc:504] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:56.217295 29636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:01:56.217528 29636 raft_consensus.cc:3032] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:56.217613 29636 raft_consensus.cc:729] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:56.217672 32096 leader_election.cc:288] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:01:56.217909 29636 consensus_queue.cc:260] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:56.218356 29636 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:56.220229 29506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:56.220417 29506 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:56.223073 29636 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:01:56.223124 29506 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:01:56.223781 32030 leader_election.cc:302] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:56.224113 9460 raft_consensus.cc:2781] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:56.224282 9460 raft_consensus.cc:686] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:01:56.224617 9460 consensus_queue.cc:237] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:56.225319 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345c7a43d2648b4b4c47e726c4d4c51" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:56.225529 29377 raft_consensus.cc:3037] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:56.229321 29377 raft_consensus.cc:2454] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:01:56.255471 29105 catalog_manager.cc:5484] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 1 to 2, leader changed from ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:56.410926 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:56.432684 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:56.447996 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:56.481289 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:56.486101 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:56.510030 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:56.517334 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:56.550952 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:01:56.560492 29180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 62995.6us
I20240501 14:01:56.658712 29506 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:56.659391 9460 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:56.668069 29636 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:56.668768 9460 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:56.703198 29374 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:56.704504 9715 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:01:57.060237 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:57.061390 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:57.065827 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:57.101472 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:57.109891 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:57.141593 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:57.141716 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:57.142501 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:57.336133 29377 consensus_queue.cc:237] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:01:57.338475 29507 raft_consensus.cc:1261] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:57.339097 8620 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:57.339475 29245 raft_consensus.cc:1261] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:57.340288 8620 consensus_queue.cc:1035] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:57.341699 9325 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:57.342535 29507 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:57.343358 29245 raft_consensus.cc:2932] T ef4b00f102874e17830f2e340255a11f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:57.350600 29102 catalog_manager.cc:5484] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:01:57.361449 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet ef4b00f102874e17830f2e340255a11f with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:57.362170 9817 tablet_replica.cc:330] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:57.362426 9817 raft_consensus.cc:2229] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:57.362680 9817 raft_consensus.cc:2258] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:57.363757 9817 ts_tablet_manager.cc:1875] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:57.371613 9817 ts_tablet_manager.cc:1888] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:57.371798 9817 log.cc:1196] T ef4b00f102874e17830f2e340255a11f P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/ef4b00f102874e17830f2e340255a11f
I20240501 14:01:57.372527 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet ef4b00f102874e17830f2e340255a11f (table rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]) successfully deleted
I20240501 14:01:57.718755 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:57.721004 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:57.755493 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:57.775136 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:57.806493 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:57.807492 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:57.816634 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:57.860123 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:58.008167 29636 consensus_queue.cc:237] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:01:58.010531 32097 raft_consensus.cc:1261] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:58.010607 29506 raft_consensus.cc:1261] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:58.011029 9703 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:58.012622 9703 consensus_queue.cc:1035] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:58.015166 9703 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:58.015870 32097 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:58.020990 29506 raft_consensus.cc:2932] T 7e0fcc86017b4589b1a960c078e7ec57 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:58.023422 29102 catalog_manager.cc:5484] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:01:58.047716 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0fcc86017b4589b1a960c078e7ec57 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:01:58.048381 9908 tablet_replica.cc:330] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:01:58.048573 9908 raft_consensus.cc:2229] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:58.048744 9908 raft_consensus.cc:2258] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:58.050071 9908 ts_tablet_manager.cc:1875] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:58.057304 9908 ts_tablet_manager.cc:1888] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:58.057520 9908 log.cc:1196] T 7e0fcc86017b4589b1a960c078e7ec57 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/7e0fcc86017b4589b1a960c078e7ec57
I20240501 14:01:58.058403 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 7e0fcc86017b4589b1a960c078e7ec57 (table rebalancer_extra_table_54 [id=5bada20aac4f41048b5257d795008db4]) successfully deleted
I20240501 14:01:58.391580 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:58.402108 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:58.439622 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:58.465991 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:58.494673 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:58.529603 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:58.569532 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:58.600872 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:58.721797 32097 consensus_queue.cc:237] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:01:58.723860 32360 raft_consensus.cc:1261] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:58.724045 29635 raft_consensus.cc:1261] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:58.724529 9715 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:58.724785 9460 consensus_queue.cc:1035] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:58.729446 9460 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:58.733492 32359 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:58.735889 29635 raft_consensus.cc:2932] T 7fce4b1a15bc48e9b8042869d5c18d16 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:58.736721 29097 catalog_manager.cc:5484] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:01:58.755831 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fce4b1a15bc48e9b8042869d5c18d16 with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:01:58.756464 10002 tablet_replica.cc:330] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:01:58.756726 10002 raft_consensus.cc:2229] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:58.757025 10002 raft_consensus.cc:2258] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:58.757603 10002 ts_tablet_manager.cc:1875] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:58.763083 10002 ts_tablet_manager.cc:1888] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:01:58.763211 10002 log.cc:1196] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/7fce4b1a15bc48e9b8042869d5c18d16
I20240501 14:01:58.763912 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 7fce4b1a15bc48e9b8042869d5c18d16 (table rebalancer_extra_table_26 [id=68f1e425c3564de6911ced6ad5be7eca]) successfully deleted
I20240501 14:01:59.099082 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:59.113355 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:59.135552 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:59.136389 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:59.206903 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:59.225736 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:59.259083 29617 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:59.289448 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:59.414321 29507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628"
dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35032
I20240501 14:01:59.414626 29507 raft_consensus.cc:595] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:59.577956 9192 raft_consensus.cc:979] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: : Instructing follower ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:01:59.578174 9192 raft_consensus.cc:1067] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Signalling peer ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:01:59.584875 29634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628"
dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
from {username='slave'} at 127.22.33.67:47041
I20240501 14:01:59.585155 29634 raft_consensus.cc:482] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:59.585281 29634 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:59.588032 29634 raft_consensus.cc:504] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:59.589773 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:01:59.589967 29377 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:59.592391 29377 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:59.593248 29507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
I20240501 14:01:59.593493 29507 raft_consensus.cc:3032] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:59.593572 29507 raft_consensus.cc:729] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:01:59.593945 29507 consensus_queue.cc:260] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:59.594489 29507 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:59.602942 29507 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:59.603431 29571 leader_election.cc:302] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:01:59.603552 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce342cbfad4419b3e2e8d3a52c4628" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:01:59.603811 31965 raft_consensus.cc:3037] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:59.604305 10101 raft_consensus.cc:2781] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:59.604451 10101 raft_consensus.cc:686] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:01:59.604709 10101 consensus_queue.cc:237] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:01:59.606442 29634 leader_election.cc:288] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:01:59.612284 29098 catalog_manager.cc:5484] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:01:59.631496 31965 raft_consensus.cc:2454] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:01:59.739845 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:59.754223 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:01:59.783154 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:59.795464 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:59.826071 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:59.843011 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:59.861833 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:59.891263 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:59.990325 29506 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:59.990921 9563 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:00.045388 31965 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:00.045938 10101 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:00.050537 29372 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:00.051419 10101 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:00.338150 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:00.362534 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:00.384837 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:00.429822 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:00.459794 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:00.477669 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:00.514950 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:00.526540 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:00.699738 32096 consensus_queue.cc:237] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:00.702315 31965 raft_consensus.cc:1261] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:00.702510 29245 raft_consensus.cc:1261] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:00.702819 10240 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:00.703195 10146 consensus_queue.cc:1035] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:00.705821 10146 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:00.706338 31965 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:00.707305 29245 raft_consensus.cc:2932] T 1eb574dc421745618ba2a11f0b3f9110 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:00.712846 29098 catalog_manager.cc:5484] T 1eb574dc421745618ba2a11f0b3f9110 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:00.726439 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb574dc421745618ba2a11f0b3f9110 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:00.740962 10253 tablet_replica.cc:330] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:00.741340 10253 raft_consensus.cc:2229] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:00.741688 10253 raft_consensus.cc:2258] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:00.742972 10253 ts_tablet_manager.cc:1875] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:00.751716 10253 ts_tablet_manager.cc:1888] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:00.751840 10253 log.cc:1196] T 1eb574dc421745618ba2a11f0b3f9110 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/1eb574dc421745618ba2a11f0b3f9110
I20240501 14:02:00.752678 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 1eb574dc421745618ba2a11f0b3f9110 (table rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]) successfully deleted
I20240501 14:02:01.144709 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:01.145347 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:01.147272 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:01.188882 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:02:01.204173 29440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 50281.5us
I20240501 14:02:01.228185 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:01.259383 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:01.273500 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:01.305790 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:01.473448 29246 consensus_queue.cc:237] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:01.476405 32359 raft_consensus.cc:1261] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:01.477082 9710 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:01.477797 32223 raft_consensus.cc:1261] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:01.478785 9710 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:01.481705 10041 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:01.482254 32359 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:01.488078 29099 catalog_manager.cc:5484] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:01.506623 32223 raft_consensus.cc:2932] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:01.530838 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a025068d40c4b388421c5e6930847c2 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:02:01.533550 10359 tablet_replica.cc:330] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:02:01.534269 10359 raft_consensus.cc:2229] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:01.535358 10359 raft_consensus.cc:2258] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:01.536248 10359 ts_tablet_manager.cc:1875] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:01.546828 10359 ts_tablet_manager.cc:1888] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:01.547111 10359 log.cc:1196] T 1a025068d40c4b388421c5e6930847c2 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/1a025068d40c4b388421c5e6930847c2
I20240501 14:02:01.548190 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 1a025068d40c4b388421c5e6930847c2 (table rebalancer_extra_table_16 [id=855664ff47f14f92a553681cf6c9e7d6]) successfully deleted
I20240501 14:02:01.803222 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:01.806586 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:01.851670 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:01.897951 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:01.938498 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:01.969307 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:02.002199 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:02.057430 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:02.244863 29372 consensus_queue.cc:237] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:02.247591 29506 raft_consensus.cc:1261] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:02.247956 32222 raft_consensus.cc:1261] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:02.248234 10286 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:02.248476 10434 consensus_queue.cc:1035] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:02.251140 10434 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:02.252355 32222 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:02.256392 29506 raft_consensus.cc:2932] T a87b1de51c4c4f909c27c27aa745fb44 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:02.266373 29097 catalog_manager.cc:5484] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 4 to 5, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:02.271132 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet a87b1de51c4c4f909c27c27aa745fb44 with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52788
I20240501 14:02:02.273321 10445 tablet_replica.cc:330] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:02:02.273494 10445 raft_consensus.cc:2229] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:02.273662 10445 raft_consensus.cc:2258] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:02.274204 10445 ts_tablet_manager.cc:1875] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:02.279711 10445 ts_tablet_manager.cc:1888] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:02.279857 10445 log.cc:1196] T a87b1de51c4c4f909c27c27aa745fb44 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/a87b1de51c4c4f909c27c27aa745fb44
I20240501 14:02:02.280750 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet a87b1de51c4c4f909c27c27aa745fb44 (table rebalance_test_table_5 [id=7de1b8153edc41bf8ce503a2aa68891d]) successfully deleted
I20240501 14:02:02.650938 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:02.654656 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:02.659897 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:02.676704 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:02.700942 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:02.717010 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:02.752890 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:02.771955 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:02.910900 31963 consensus_queue.cc:237] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:02.913442 29634 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:02.913432 32223 raft_consensus.cc:1261] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:02.913972 10496 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:02.914222 10223 consensus_queue.cc:1035] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:02.916678 10223 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:02.917354 29634 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:02.925684 29098 catalog_manager.cc:5484] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:02.936084 32223 raft_consensus.cc:2932] T b5aa71ecef4d419eb36c7777790e6f08 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:02.952505 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet b5aa71ecef4d419eb36c7777790e6f08 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:02.953238 10544 tablet_replica.cc:330] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:02.953922 10544 raft_consensus.cc:2229] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:02.954741 10544 raft_consensus.cc:2258] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:02.956761 10544 ts_tablet_manager.cc:1875] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:02.963202 10544 ts_tablet_manager.cc:1888] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 14:02:02.963335 10544 log.cc:1196] T b5aa71ecef4d419eb36c7777790e6f08 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/b5aa71ecef4d419eb36c7777790e6f08
I20240501 14:02:02.964090 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet b5aa71ecef4d419eb36c7777790e6f08 (table rebalancer_extra_table_20 [id=9d885186e6754868b055c9583210b3f4]) successfully deleted
W20240501 14:02:03.229594 29572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 63181.5us
I20240501 14:02:03.246747 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:03.272194 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:03.341194 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:03.342043 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:03.348841 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:03.353821 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:03.376787 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:03.433954 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:03.581524 29507 consensus_queue.cc:237] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:03.583492 32097 raft_consensus.cc:1261] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:03.583513 29634 raft_consensus.cc:1261] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:03.584007 10605 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:03.584261 10411 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:03.586687 10605 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:03.588083 29634 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:03.588403 32097 raft_consensus.cc:2932] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:03.594195 29102 catalog_manager.cc:5484] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 4 to 5, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:03.612113 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c02994dfef4a10a87b1f47d3ab6e3b with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:03.614339 10630 tablet_replica.cc:330] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:03.614672 10630 raft_consensus.cc:2229] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:03.614944 10630 raft_consensus.cc:2258] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:03.615911 10630 ts_tablet_manager.cc:1875] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:03.621310 10630 ts_tablet_manager.cc:1888] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:03.621444 10630 log.cc:1196] T 21c02994dfef4a10a87b1f47d3ab6e3b P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/21c02994dfef4a10a87b1f47d3ab6e3b
I20240501 14:02:03.622135 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 21c02994dfef4a10a87b1f47d3ab6e3b (table rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]) successfully deleted
I20240501 14:02:03.879751 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:03.905810 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:03.972484 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:03.985728 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:04.034008 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:04.039983 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:04.072350 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:04.079079 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:04.272841 32097 consensus_queue.cc:237] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:02:04.274863 29374 raft_consensus.cc:1261] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:04.275141 32360 raft_consensus.cc:1261] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:04.275381 10146 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:04.275635 10427 consensus_queue.cc:1035] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:04.282351 10427 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:02:04.283187 29374 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:02:04.285171 32360 raft_consensus.cc:2932] T 815ce94772f2454ea755d33dd04a87d2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:02:04.295574 29102 catalog_manager.cc:5484] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } }
I20240501 14:02:04.310060 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 815ce94772f2454ea755d33dd04a87d2 with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:02:04.311061 10727 tablet_replica.cc:330] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:02:04.311326 10727 raft_consensus.cc:2229] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:04.311512 10727 raft_consensus.cc:2258] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:04.312274 10727 ts_tablet_manager.cc:1875] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:04.321254 10727 ts_tablet_manager.cc:1888] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:04.321552 10727 log.cc:1196] T 815ce94772f2454ea755d33dd04a87d2 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/815ce94772f2454ea755d33dd04a87d2
I20240501 14:02:04.322547 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 815ce94772f2454ea755d33dd04a87d2 (table rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]) successfully deleted
I20240501 14:02:04.362027 32359 consensus_queue.cc:237] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:04.364820 31965 raft_consensus.cc:1261] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.365016 29374 raft_consensus.cc:1261] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:04.365710 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b12b60d5bf304c28ae2041629b8d5bdd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:04.365731 10710 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.366112 10431 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.369071 10431 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.369738 29374 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.373207 31965 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.378958 29099 catalog_manager.cc:5484] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.414983 32222 consensus_queue.cc:237] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:04.418390 29374 raft_consensus.cc:1261] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.419035 10668 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:02:04.422098 32163 consensus_peers.cc:485] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c23d0b871aff4ce3850e45ae31097c03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:04.421890 32359 raft_consensus.cc:1261] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.427913 10764 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.431114 10668 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:04.436321 29374 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:04.439474 32359 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:04.445711 29103 catalog_manager.cc:5484] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:04.523537 32359 consensus_queue.cc:237] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:04.526273 31965 raft_consensus.cc:1261] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.526389 32222 raft_consensus.cc:1261] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.527050 10710 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:02:04.527024 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd7a7501266946dca6bc838d10524c4a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:04.529078 10710 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.559290 10710 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.561332 32222 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.563717 31965 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.566783 29099 catalog_manager.cc:5484] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:04.612478 32096 consensus_queue.cc:237] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
W20240501 14:02:04.626525 32030 consensus_peers.cc:485] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768543d354744b44acdebb9a95a1579b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:04.626621 32222 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.626621 31965 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:04.627166 10427 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.627468 10146 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:04.630820 10146 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:04.631459 31965 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:04.635830 32222 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:04.639628 29103 catalog_manager.cc:5484] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:04.692756 29094 maintenance_manager.cc:419] P 589b051e6abe4fa38c771a41c6df7284: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033388
I20240501 14:02:04.746117 10820 ts_tablet_manager.cc:902] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:04.761164 10820 tablet_copy_client.cc:308] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:02:04.762076 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet b12b60d5bf304c28ae2041629b8d5bdd from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:51381)
I20240501 14:02:04.762241 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet b12b60d5bf304c28ae2041629b8d5bdd from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:51381: session id = 522f4ff5e9084cbf8462196533c7d7aa-b12b60d5bf304c28ae2041629b8d5bdd
I20240501 14:02:04.766224 32383 tablet_copy_source_session.cc:205] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:04.768080 29093 maintenance_manager.cc:643] P 589b051e6abe4fa38c771a41c6df7284: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.054s sys 0.012s Metrics: {"bytes_written":93120,"cfile_init":1,"dirs.queue_time_us":1609,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":748,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1776,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":257,"spinlock_wait_cycles":52608,"thread_start_us":1502,"threads_started":1}
I20240501 14:02:04.769542 10820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12b60d5bf304c28ae2041629b8d5bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:04.770186 29094 maintenance_manager.cc:419] P 589b051e6abe4fa38c771a41c6df7284: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 30826 bytes on disk
I20240501 14:02:04.774834 29093 maintenance_manager.cc:643] P 589b051e6abe4fa38c771a41c6df7284: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240501 14:02:04.777648 10820 tablet_copy_client.cc:783] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:04.777884 10820 tablet_copy_client.cc:647] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:04.779609 10820 tablet_copy_client.cc:523] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:04.783380 10820 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:04.808244 10820 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:04.808765 10820 tablet_bootstrap.cc:492] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:04.809157 10820 ts_tablet_manager.cc:1367] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.000s
I20240501 14:02:04.810209 10820 raft_consensus.cc:348] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:04.810411 10820 raft_consensus.cc:729] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:04.810618 10820 consensus_queue.cc:260] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:04.811062 10820 ts_tablet_manager.cc:1398] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:04.811686 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-b12b60d5bf304c28ae2041629b8d5bdd received from {username='slave'} at 127.22.33.67:51381
I20240501 14:02:04.811815 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-b12b60d5bf304c28ae2041629b8d5bdd on tablet b12b60d5bf304c28ae2041629b8d5bdd with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:04.837708 10835 ts_tablet_manager.cc:902] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:04.839212 10835 tablet_copy_client.cc:308] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:02:04.839987 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet c23d0b871aff4ce3850e45ae31097c03 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:60545)
I20240501 14:02:04.840193 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet c23d0b871aff4ce3850e45ae31097c03 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:60545: session id = 0a697f3a1ee544daa7073814ab845d58-c23d0b871aff4ce3850e45ae31097c03
I20240501 14:02:04.845155 32248 tablet_copy_source_session.cc:205] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:04.850234 10835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23d0b871aff4ce3850e45ae31097c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:04.863688 10835 tablet_copy_client.cc:783] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:04.864012 10835 tablet_copy_client.cc:647] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:04.901551 10835 tablet_copy_client.cc:523] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:04.908293 10835 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:02:04.973243 10820 ts_tablet_manager.cc:902] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:05.002662 29312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 63283.7us
I20240501 14:02:05.026005 10820 tablet_copy_client.cc:308] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:02:05.027022 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet cd7a7501266946dca6bc838d10524c4a from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:51381)
I20240501 14:02:05.027176 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet cd7a7501266946dca6bc838d10524c4a from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:51381: session id = 522f4ff5e9084cbf8462196533c7d7aa-cd7a7501266946dca6bc838d10524c4a
I20240501 14:02:05.028200 10835 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:05.028614 10835 tablet_bootstrap.cc:492] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:02:05.028869 10835 ts_tablet_manager.cc:1367] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.121s user 0.031s sys 0.023s
I20240501 14:02:05.029569 10835 raft_consensus.cc:348] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:05.029772 10835 raft_consensus.cc:729] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:02:05.030165 32383 tablet_copy_source_session.cc:205] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:05.031533 10820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd7a7501266946dca6bc838d10524c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:05.031462 10835 consensus_queue.cc:260] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:05.033502 10835 ts_tablet_manager.cc:1398] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:05.040694 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-c23d0b871aff4ce3850e45ae31097c03 received from {username='slave'} at 127.22.33.70:60545
I20240501 14:02:05.040851 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-c23d0b871aff4ce3850e45ae31097c03 on tablet c23d0b871aff4ce3850e45ae31097c03 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:02:05.069309 10820 tablet_copy_client.cc:783] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:05.069579 10820 tablet_copy_client.cc:647] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:05.073369 10820 tablet_copy_client.cc:523] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:05.077919 10820 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:05.132611 10820 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:05.144223 10885 ts_tablet_manager.cc:902] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:05.147179 10820 tablet_bootstrap.cc:492] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:05.147832 10820 ts_tablet_manager.cc:1367] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.008s
I20240501 14:02:05.155390 10820 raft_consensus.cc:348] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:05.157100 10820 raft_consensus.cc:729] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:05.157832 10820 consensus_queue.cc:260] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:05.159188 10820 ts_tablet_manager.cc:1398] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:02:05.166231 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-cd7a7501266946dca6bc838d10524c4a received from {username='slave'} at 127.22.33.67:51381
I20240501 14:02:05.166385 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-cd7a7501266946dca6bc838d10524c4a on tablet cd7a7501266946dca6bc838d10524c4a with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:05.228696 10885 tablet_copy_client.cc:308] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:05.229758 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 768543d354744b44acdebb9a95a1579b from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57709)
I20240501 14:02:05.229920 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 768543d354744b44acdebb9a95a1579b from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57709: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-768543d354744b44acdebb9a95a1579b
I20240501 14:02:05.232625 32117 tablet_copy_source_session.cc:205] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:05.276181 10885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768543d354744b44acdebb9a95a1579b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:05.286124 10885 tablet_copy_client.cc:783] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:05.286393 10885 tablet_copy_client.cc:647] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:05.293299 10885 tablet_copy_client.cc:523] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:05.294276 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:05.297173 10885 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:05.339802 29507 raft_consensus.cc:1203] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:05.339838 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:05.385483 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:05.398938 10885 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:05.399412 10885 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:05.399716 10885 ts_tablet_manager.cc:1367] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.003s
I20240501 14:02:05.406987 10885 raft_consensus.cc:348] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:05.407243 10885 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:05.407495 10885 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:05.409739 10885 ts_tablet_manager.cc:1398] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20240501 14:02:05.411577 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-768543d354744b44acdebb9a95a1579b received from {username='slave'} at 127.22.33.68:57709
I20240501 14:02:05.411726 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-768543d354744b44acdebb9a95a1579b on tablet 768543d354744b44acdebb9a95a1579b with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:05.429286 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:05.443753 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:05.453675 32096 raft_consensus.cc:1203] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:05.462234 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:05.466630 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:05.482580 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:05.517742 29507 raft_consensus.cc:1203] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:05.607465 29635 raft_consensus.cc:1203] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:05.874614 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:05.921936 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:05.951359 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:05.958302 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:05.980923 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:05.991611 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:06.045545 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:06.045888 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:06.197489 32096 consensus_queue.cc:237] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:06.199568 29507 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:06.200487 32223 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:06.200568 10427 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:06.201750 10427 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:06.204584 10697 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.208196 32222 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.210785 29101 catalog_manager.cc:5484] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.224035 29507 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.228904 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:06.236282 10999 tablet_replica.cc:330] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:06.236544 10999 raft_consensus.cc:2229] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:06.236793 10999 raft_consensus.cc:2258] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:06.237480 10999 ts_tablet_manager.cc:1875] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:06.245174 10999 ts_tablet_manager.cc:1888] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:06.249068 10999 log.cc:1196] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/54bdc21d1dc644e7ac44f5b9ad39ab32
I20240501 14:02:06.249943 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 (table rebalancer_extra_table_2 [id=0240ab2e245d4c88b27a887c75659987]) successfully deleted
I20240501 14:02:06.514031 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:06.568303 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:06.575023 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:06.610030 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:06.642724 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:06.663769 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:06.691457 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:06.721099 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:06.875685 32361 consensus_queue.cc:237] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:06.877954 29372 raft_consensus.cc:1261] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:06.878258 29506 raft_consensus.cc:1261] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:06.878798 10710 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:06.879523 10710 consensus_queue.cc:1035] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:06.882795 10710 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.885370 29372 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.894217 29102 catalog_manager.cc:5484] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.899163 29506 raft_consensus.cc:2932] T b12b60d5bf304c28ae2041629b8d5bdd P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:06.909562 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet b12b60d5bf304c28ae2041629b8d5bdd with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:02:06.917891 11091 tablet_replica.cc:330] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:02:06.920342 11091 raft_consensus.cc:2229] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:06.920655 11091 raft_consensus.cc:2258] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:06.921730 11091 ts_tablet_manager.cc:1875] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:06.930471 11091 ts_tablet_manager.cc:1888] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:06.930708 11091 log.cc:1196] T b12b60d5bf304c28ae2041629b8d5bdd P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/b12b60d5bf304c28ae2041629b8d5bdd
I20240501 14:02:06.931614 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet b12b60d5bf304c28ae2041629b8d5bdd (table rebalancer_extra_table_18 [id=0457d1d1f49c42b2b7a2d3bdd9bbd07b]) successfully deleted
I20240501 14:02:07.300901 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:07.337566 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:07.339213 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:07.356020 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:07.383782 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:07.385835 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:07.492596 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:07.500115 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:07.622483 32096 consensus_queue.cc:237] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:07.624675 29372 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:07.624791 29506 raft_consensus.cc:1261] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:07.625737 10697 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:07.626248 10697 consensus_queue.cc:1035] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:07.631317 10427 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:07.635200 29372 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:07.639003 29098 catalog_manager.cc:5484] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 4 to 6, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:07.645839 29506 raft_consensus.cc:2932] T f345c7a43d2648b4b4c47e726c4d4c51 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:07.652101 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet f345c7a43d2648b4b4c47e726c4d4c51 with delete_type TABLET_DATA_TOMBSTONED (TS ecc8bf0965164f199fb0bfa3d7b47c0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52788
I20240501 14:02:07.653585 11181 tablet_replica.cc:330] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: stopping tablet replica
I20240501 14:02:07.653825 11181 raft_consensus.cc:2229] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:07.654088 11181 raft_consensus.cc:2258] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:07.655292 11181 ts_tablet_manager.cc:1875] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:07.663903 11181 ts_tablet_manager.cc:1888] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:02:07.664083 11181 log.cc:1196] T f345c7a43d2648b4b4c47e726c4d4c51 P ecc8bf0965164f199fb0bfa3d7b47c0e: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/wals/f345c7a43d2648b4b4c47e726c4d4c51
I20240501 14:02:07.665030 29089 catalog_manager.cc:4830] TS ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): tablet f345c7a43d2648b4b4c47e726c4d4c51 (table rebalance_test_table_7 [id=0da56e35536a4d32a2a1284a78195d3b]) successfully deleted
I20240501 14:02:08.009342 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:08.049993 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:08.113835 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:08.130667 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:08.170907 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:08.206851 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:08.207325 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:08.210716 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:02:08.293066 29441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 75015.3us
I20240501 14:02:08.387677 29634 consensus_queue.cc:237] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:08.390185 29372 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:08.390364 31964 raft_consensus.cc:1261] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:08.390748 11064 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:08.391109 10884 consensus_queue.cc:1035] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:08.394376 11064 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:08.394801 31964 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:08.394991 29372 raft_consensus.cc:2932] T 77ce342cbfad4419b3e2e8d3a52c4628 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:08.401332 29101 catalog_manager.cc:5484] T 77ce342cbfad4419b3e2e8d3a52c4628 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 4 to 6, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:08.407467 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ce342cbfad4419b3e2e8d3a52c4628 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41216
I20240501 14:02:08.417486 11278 tablet_replica.cc:330] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:02:08.417774 11278 raft_consensus.cc:2229] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:08.417982 11278 raft_consensus.cc:2258] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:08.418807 11278 ts_tablet_manager.cc:1875] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:08.425588 11278 ts_tablet_manager.cc:1888] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:02:08.425725 11278 log.cc:1196] T 77ce342cbfad4419b3e2e8d3a52c4628 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/77ce342cbfad4419b3e2e8d3a52c4628
I20240501 14:02:08.426996 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 77ce342cbfad4419b3e2e8d3a52c4628 (table rebalance_test_table_9 [id=841e307cb41f45debe275139822553e6]) successfully deleted
I20240501 14:02:08.745513 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:08.752120 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:08.785275 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:08.785878 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:08.821985 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:08.858824 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:08.860714 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:08.873775 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:09.059870 32360 consensus_queue.cc:237] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:09.061882 32224 raft_consensus.cc:1261] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:09.061883 29506 raft_consensus.cc:1261] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:09.062367 11045 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:09.062638 10710 consensus_queue.cc:1035] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:09.066196 10710 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.069111 32224 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.069182 29506 raft_consensus.cc:2932] T cd7a7501266946dca6bc838d10524c4a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.071766 29100 catalog_manager.cc:5484] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.087920 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet cd7a7501266946dca6bc838d10524c4a with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37626
I20240501 14:02:09.088665 11364 tablet_replica.cc:330] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:02:09.088860 11364 raft_consensus.cc:2229] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:09.089123 11364 raft_consensus.cc:2258] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:09.089920 11364 ts_tablet_manager.cc:1875] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:09.095690 11364 ts_tablet_manager.cc:1888] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:09.096190 11364 log.cc:1196] T cd7a7501266946dca6bc838d10524c4a P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/cd7a7501266946dca6bc838d10524c4a
I20240501 14:02:09.097136 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet cd7a7501266946dca6bc838d10524c4a (table rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]) successfully deleted
I20240501 14:02:09.368250 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:09.380373 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:09.416067 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:09.435047 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:09.442718 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:09.455090 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:09.479211 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:09.515200 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:09.682960 32096 consensus_queue.cc:237] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:09.685410 31963 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:09.685499 29635 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:09.685972 11331 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:09.686635 10697 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:09.688681 10697 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.690122 29634 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.692389 31963 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.694928 29105 catalog_manager.cc:5484] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.721684 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 768543d354744b44acdebb9a95a1579b with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:02:09.725418 11453 tablet_replica.cc:330] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:02:09.725694 11453 raft_consensus.cc:2229] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:09.725978 11453 raft_consensus.cc:2258] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:09.726704 11453 ts_tablet_manager.cc:1875] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:09.732122 11453 ts_tablet_manager.cc:1888] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:09.732240 11453 log.cc:1196] T 768543d354744b44acdebb9a95a1579b P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/768543d354744b44acdebb9a95a1579b
I20240501 14:02:09.733139 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet 768543d354744b44acdebb9a95a1579b (table rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]) successfully deleted
I20240501 14:02:09.788633 29377 consensus_queue.cc:237] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:09.791077 32360 raft_consensus.cc:1261] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:09.791307 29312 consensus_peers.cc:485] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d8239317634874b1ffb1ba239bb9ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.791364 29634 raft_consensus.cc:1261] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:09.791872 10867 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.792444 10867 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.795307 11483 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:09.795846 32360 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:09.800324 29634 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:09.802222 29105 catalog_manager.cc:5484] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:09.877034 29377 consensus_queue.cc:237] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:09.881678 32224 raft_consensus.cc:1261] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:09.881778 31963 raft_consensus.cc:1261] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:09.882406 29310 consensus_peers.cc:485] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db862a636e5a427085a2fa31f6fe8050. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.882454 11486 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.882733 11012 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.886397 11486 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.887060 32224 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.890755 31963 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.893326 29105 catalog_manager.cc:5484] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:09.941890 29634 consensus_queue.cc:237] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:09.944622 32224 raft_consensus.cc:1261] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:09.945276 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a1690e3a76644859a0711e305b8f946. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.945504 32360 raft_consensus.cc:1261] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:09.945586 10884 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.946501 10884 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:09.949889 11272 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.953521 32359 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.956710 32224 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:09.957587 29105 catalog_manager.cc:5484] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:10.005259 29377 consensus_queue.cc:237] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:10.007582 32096 raft_consensus.cc:1261] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:10.007611 32359 raft_consensus.cc:1261] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:10.008266 29310 consensus_peers.cc:485] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68f3644410a24fc1a9457ff46a10c180. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:10.008294 10867 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:10.008558 11486 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:10.013154 10867 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:10.013902 32096 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:10.020936 29099 catalog_manager.cc:5484] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:10.021551 32359 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:10.223867 11577 ts_tablet_manager.cc:902] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:10.269058 11577 tablet_copy_client.cc:308] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:10.269955 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet b0d8239317634874b1ffb1ba239bb9ad from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:38381)
I20240501 14:02:10.270136 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet b0d8239317634874b1ffb1ba239bb9ad from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:38381: session id = 0a697f3a1ee544daa7073814ab845d58-b0d8239317634874b1ffb1ba239bb9ad
I20240501 14:02:10.273047 29397 tablet_copy_source_session.cc:205] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:10.274657 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d8239317634874b1ffb1ba239bb9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:10.287380 11577 tablet_copy_client.cc:783] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:10.287642 11577 tablet_copy_client.cc:647] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:10.289192 11577 tablet_copy_client.cc:523] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:10.292935 11577 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:02:10.314781 11577 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:10.315177 11577 tablet_bootstrap.cc:492] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:02:10.315423 11577 ts_tablet_manager.cc:1367] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.019s
I20240501 14:02:10.316255 11577 raft_consensus.cc:348] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:10.316502 11577 raft_consensus.cc:729] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:02:10.316849 11577 consensus_queue.cc:260] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:10.317592 11577 ts_tablet_manager.cc:1398] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:02:10.318291 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-b0d8239317634874b1ffb1ba239bb9ad received from {username='slave'} at 127.22.33.70:38381
I20240501 14:02:10.318459 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-b0d8239317634874b1ffb1ba239bb9ad on tablet b0d8239317634874b1ffb1ba239bb9ad with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:02:10.347244 11592 ts_tablet_manager.cc:902] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:10.364837 11592 tablet_copy_client.cc:308] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:10.366159 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet db862a636e5a427085a2fa31f6fe8050 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:35721)
I20240501 14:02:10.366359 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet db862a636e5a427085a2fa31f6fe8050 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:35721: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-db862a636e5a427085a2fa31f6fe8050
I20240501 14:02:10.368911 29397 tablet_copy_source_session.cc:205] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:10.371110 11592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db862a636e5a427085a2fa31f6fe8050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:10.378458 11592 tablet_copy_client.cc:783] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:10.378738 11592 tablet_copy_client.cc:647] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:10.391361 11592 tablet_copy_client.cc:523] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:10.395287 11592 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:10.401438 11592 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:10.401885 11592 tablet_bootstrap.cc:492] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:10.402153 11592 ts_tablet_manager.cc:1367] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:02:10.402937 11592 raft_consensus.cc:348] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:10.403183 11592 raft_consensus.cc:729] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:10.403445 11592 consensus_queue.cc:260] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:10.404073 11592 ts_tablet_manager.cc:1398] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:10.404954 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-db862a636e5a427085a2fa31f6fe8050 received from {username='slave'} at 127.22.33.68:35721
I20240501 14:02:10.405102 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-db862a636e5a427085a2fa31f6fe8050 on tablet db862a636e5a427085a2fa31f6fe8050 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:10.420222 11592 ts_tablet_manager.cc:902] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:10.462297 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:10.463449 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:10.473505 11592 tablet_copy_client.cc:308] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:10.473735 11614 ts_tablet_manager.cc:902] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:10.482893 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 68f3644410a24fc1a9457ff46a10c180 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:35721)
I20240501 14:02:10.483064 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 68f3644410a24fc1a9457ff46a10c180 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:35721: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-68f3644410a24fc1a9457ff46a10c180
I20240501 14:02:10.485877 29397 tablet_copy_source_session.cc:205] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:10.487390 11592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f3644410a24fc1a9457ff46a10c180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:10.499008 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:10.520172 11614 tablet_copy_client.cc:308] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:02:10.520803 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 8a1690e3a76644859a0711e305b8f946 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:47041)
I20240501 14:02:10.521008 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 8a1690e3a76644859a0711e305b8f946 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:47041: session id = 522f4ff5e9084cbf8462196533c7d7aa-8a1690e3a76644859a0711e305b8f946
I20240501 14:02:10.521477 11592 tablet_copy_client.cc:783] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:10.521801 11592 tablet_copy_client.cc:647] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:10.524092 29657 tablet_copy_source_session.cc:205] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:10.525817 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:10.527928 11592 tablet_copy_client.cc:523] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:10.529624 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1690e3a76644859a0711e305b8f946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:10.531471 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:10.566812 11592 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:10.596793 11614 tablet_copy_client.cc:783] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:10.597110 11614 tablet_copy_client.cc:647] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:10.605127 11614 tablet_copy_client.cc:523] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:10.611160 11614 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:10.617121 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:10.643965 11592 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:10.644436 11592 tablet_bootstrap.cc:492] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:10.644721 11592 ts_tablet_manager.cc:1367] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.002s
I20240501 14:02:10.645541 11592 raft_consensus.cc:348] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:10.645776 11592 raft_consensus.cc:729] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:10.646024 11592 consensus_queue.cc:260] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:10.646673 11592 ts_tablet_manager.cc:1398] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:10.675509 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-68f3644410a24fc1a9457ff46a10c180 received from {username='slave'} at 127.22.33.68:35721
I20240501 14:02:10.675673 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-68f3644410a24fc1a9457ff46a10c180 on tablet 68f3644410a24fc1a9457ff46a10c180 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:10.676545 32096 raft_consensus.cc:1203] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:10.682694 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:10.686650 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:10.698530 11614 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:10.699085 11614 tablet_bootstrap.cc:492] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:10.699429 11614 ts_tablet_manager.cc:1367] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.019s
I20240501 14:02:10.700280 11614 raft_consensus.cc:348] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:10.700731 11614 raft_consensus.cc:729] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:10.701164 11614 consensus_queue.cc:260] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:10.703984 11614 ts_tablet_manager.cc:1398] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:10.704643 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-8a1690e3a76644859a0711e305b8f946 received from {username='slave'} at 127.22.33.67:47041
I20240501 14:02:10.704774 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-8a1690e3a76644859a0711e305b8f946 on tablet 8a1690e3a76644859a0711e305b8f946 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:10.834368 29635 raft_consensus.cc:1203] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:10.852512 29507 raft_consensus.cc:1203] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:10.951493 29635 raft_consensus.cc:1203] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:11.124841 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:11.137837 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:11.169317 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:11.202116 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:11.208576 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:11.230437 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:11.246551 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:11.285626 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:11.430169 29374 consensus_queue.cc:237] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:11.432061 29635 raft_consensus.cc:1261] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:11.432080 31963 raft_consensus.cc:1261] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:11.432611 10867 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:11.432847 11486 consensus_queue.cc:1035] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:11.435489 10867 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:11.436062 31963 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:11.443276 29099 catalog_manager.cc:5484] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:11.464232 29635 raft_consensus.cc:2932] T db862a636e5a427085a2fa31f6fe8050 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:11.482290 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet db862a636e5a427085a2fa31f6fe8050 with delete_type TABLET_DATA_TOMBSTONED (TS 6a446ac802a7457d9f27076781f21d79 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34442
I20240501 14:02:11.496073 11727 tablet_replica.cc:330] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: stopping tablet replica
I20240501 14:02:11.496455 11727 raft_consensus.cc:2229] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:11.496855 11727 raft_consensus.cc:2258] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:11.497743 11727 ts_tablet_manager.cc:1875] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:11.505542 11727 ts_tablet_manager.cc:1888] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:11.505795 11727 log.cc:1196] T db862a636e5a427085a2fa31f6fe8050 P 6a446ac802a7457d9f27076781f21d79: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/wals/db862a636e5a427085a2fa31f6fe8050
I20240501 14:02:11.506668 29090 catalog_manager.cc:4830] TS 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): tablet db862a636e5a427085a2fa31f6fe8050 (table rebalancer_extra_table_30 [id=a2482224ffbf41dc90c53c574cbfa03a]) successfully deleted
I20240501 14:02:11.790882 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:11.811738 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:11.824029 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 14:02:11.874639 32294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 73231us
I20240501 14:02:11.879424 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:11.901839 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:11.923343 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:11.939615 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:02:11.954013 29571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 58329us
I20240501 14:02:11.979848 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:12.123143 29377 consensus_queue.cc:237] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:12.125241 29635 raft_consensus.cc:1261] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:12.125376 32097 raft_consensus.cc:1261] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:12.125842 10867 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:12.126127 11486 consensus_queue.cc:1035] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:12.128784 11486 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.129397 32097 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.133239 29635 raft_consensus.cc:2932] T b0d8239317634874b1ffb1ba239bb9ad P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.137014 29102 catalog_manager.cc:5484] T b0d8239317634874b1ffb1ba239bb9ad P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.146685 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d8239317634874b1ffb1ba239bb9ad with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:12.147347 11819 tablet_replica.cc:330] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:12.147603 11819 raft_consensus.cc:2229] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:12.147862 11819 raft_consensus.cc:2258] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:12.148573 11819 ts_tablet_manager.cc:1875] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:12.155562 11819 ts_tablet_manager.cc:1888] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:12.155781 11819 log.cc:1196] T b0d8239317634874b1ffb1ba239bb9ad P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/b0d8239317634874b1ffb1ba239bb9ad
I20240501 14:02:12.156596 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet b0d8239317634874b1ffb1ba239bb9ad (table rebalancer_extra_table_40 [id=920d1d3b8a0f429fa767e88b10d60f70]) successfully deleted
I20240501 14:02:12.487267 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:12.498729 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:12.503934 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:12.525576 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:12.546742 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:12.568804 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:12.611379 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:12.656139 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:12.791524 32223 consensus_queue.cc:237] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:12.793413 32097 raft_consensus.cc:1261] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:12.793475 32361 raft_consensus.cc:1261] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:12.793978 11862 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:12.794344 11066 consensus_queue.cc:1035] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:12.797199 11862 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.797830 32097 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.805409 29102 catalog_manager.cc:5484] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.807322 32361 raft_consensus.cc:2932] T c23d0b871aff4ce3850e45ae31097c03 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:12.831667 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet c23d0b871aff4ce3850e45ae31097c03 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:12.852635 11903 tablet_replica.cc:330] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:12.853206 11903 raft_consensus.cc:2229] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:12.853664 11903 raft_consensus.cc:2258] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:12.854733 11903 ts_tablet_manager.cc:1875] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:12.875362 11903 ts_tablet_manager.cc:1888] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:12.875581 11903 log.cc:1196] T c23d0b871aff4ce3850e45ae31097c03 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/c23d0b871aff4ce3850e45ae31097c03
I20240501 14:02:12.878595 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet c23d0b871aff4ce3850e45ae31097c03 (table rebalancer_extra_table_46 [id=b1930d21be4c43ca995bd2cb7110bb40]) successfully deleted
I20240501 14:02:13.197134 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:13.226912 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:13.300020 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:13.307854 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:13.312389 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:13.357841 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:13.415997 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:13.430343 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:13.575130 29374 consensus_queue.cc:237] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:13.577009 32361 raft_consensus.cc:1261] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:13.577059 29634 raft_consensus.cc:1261] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:13.577543 11486 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:13.577844 11815 consensus_queue.cc:1035] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:13.581005 11815 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.581979 32361 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.581941 29634 raft_consensus.cc:2932] T 68f3644410a24fc1a9457ff46a10c180 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.589352 29099 catalog_manager.cc:5484] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.600833 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f3644410a24fc1a9457ff46a10c180 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:02:13.618134 12005 tablet_replica.cc:330] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:02:13.618546 12005 raft_consensus.cc:2229] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:13.618847 12005 raft_consensus.cc:2258] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:13.619737 12005 ts_tablet_manager.cc:1875] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:13.626080 12005 ts_tablet_manager.cc:1888] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:13.626219 12005 log.cc:1196] T 68f3644410a24fc1a9457ff46a10c180 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/68f3644410a24fc1a9457ff46a10c180
I20240501 14:02:13.628170 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet 68f3644410a24fc1a9457ff46a10c180 (table rebalancer_extra_table_24 [id=b049cac3b2274c15aa1d1e071488e9bd]) successfully deleted
I20240501 14:02:13.678117 29374 consensus_queue.cc:237] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:13.680855 29244 raft_consensus.cc:1261] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:13.680902 29507 raft_consensus.cc:1261] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:13.681059 29311 consensus_peers.cc:485] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2c63489706e4d068adde1872f451ad2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:13.681458 11486 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:13.681802 11997 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:13.690114 11997 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:13.690757 29507 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:13.694203 29244 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:13.697621 29105 catalog_manager.cc:5484] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:13.737123 32097 consensus_queue.cc:237] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:13.739585 29507 raft_consensus.cc:1261] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:13.739792 32361 raft_consensus.cc:1261] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:13.740118 11597 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:02:13.740403 32030 consensus_peers.cc:485] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7cd18adcd844f4be7ddd343cda3df4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:13.740710 11597 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:13.743254 12052 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.743857 32361 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.749961 29507 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.750814 29105 catalog_manager.cc:5484] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:13.791728 31965 consensus_queue.cc:237] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:13.794476 32224 raft_consensus.cc:1261] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:13.795282 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba3bd81448244b5bbdcd9eb8d23aee03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:13.795696 32096 raft_consensus.cc:1261] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:13.797245 10496 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:13.797652 11962 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:13.816799 10496 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:13.821355 32223 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:13.830618 29096 catalog_manager.cc:5484] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:13.833081 32096 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:13.879957 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(578c3f4c5a7c4e2c962b64736195f0bc): perf score=0.033340
I20240501 14:02:13.936543 32096 consensus_queue.cc:237] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:13.938522 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(8427dee24ad04247b23b24570c73c92a): perf score=0.033371
I20240501 14:02:13.980398 32359 raft_consensus.cc:1261] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:13.980947 32224 raft_consensus.cc:1261] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:13.982637 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0132590f8ed5423f983c53d9c19f95fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:13.982959 12052 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:13.983574 12052 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:14.011698 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(578c3f4c5a7c4e2c962b64736195f0bc) complete. Timing: real 0.131s user 0.060s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":398,"dirs.queue_time_us":8374,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1361,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1049,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21738,"threads_started":3,"wal-append.queue_time_us":13272}
I20240501 14:02:14.015869 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc): 4834 bytes on disk
I20240501 14:02:14.016876 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(578c3f4c5a7c4e2c962b64736195f0bc): perf score=0.033388
I20240501 14:02:14.018404 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.031251 11685 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:14.035784 32222 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:14.041441 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(21c02994dfef4a10a87b1f47d3ab6e3b): perf score=0.033386
I20240501 14:02:14.047406 32361 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:14.060590 29096 catalog_manager.cc:5484] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:14.196895 12110 ts_tablet_manager.cc:902] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:14.213781 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(8427dee24ad04247b23b24570c73c92a) complete. Timing: real 0.275s user 0.060s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":27857,"dirs.queue_time_us":10751,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1054,"drs_written":1,"lbm_read_time_us":179,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1189,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58218,"threads_started":3,"wal-append.queue_time_us":16915}
I20240501 14:02:14.217844 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(8427dee24ad04247b23b24570c73c92a): 4581 bytes on disk
I20240501 14:02:14.222626 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(8427dee24ad04247b23b24570c73c92a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.225229 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(21c02994dfef4a10a87b1f47d3ab6e3b): perf score=0.033445
I20240501 14:02:14.269634 12110 tablet_copy_client.cc:308] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:14.270455 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet c2c63489706e4d068adde1872f451ad2 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:49865)
I20240501 14:02:14.270617 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet c2c63489706e4d068adde1872f451ad2 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:49865: session id = 6edd6a3430d44c38a19c2804e555059f-c2c63489706e4d068adde1872f451ad2
I20240501 14:02:14.275171 29397 tablet_copy_source_session.cc:205] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:14.279050 12110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c63489706e4d068adde1872f451ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.290797 12110 tablet_copy_client.cc:783] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:14.298924 12110 tablet_copy_client.cc:647] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:14.303347 12110 tablet_copy_client.cc:523] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:14.306967 12110 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:02:14.323738 12123 ts_tablet_manager.cc:902] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:14.346573 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(578c3f4c5a7c4e2c962b64736195f0bc) complete. Timing: real 0.329s user 0.058s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":356,"dirs.queue_time_us":8329,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1115,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1063,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62853,"threads_started":3,"wal-append.queue_time_us":26959}
I20240501 14:02:14.350718 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc): 4834 bytes on disk
I20240501 14:02:14.365943 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.368743 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(8427dee24ad04247b23b24570c73c92a): perf score=0.033481
I20240501 14:02:14.404358 12128 ts_tablet_manager.cc:902] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:14.412715 12123 tablet_copy_client.cc:308] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:14.413691 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 9c7cd18adcd844f4be7ddd343cda3df4 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57709)
I20240501 14:02:14.454365 12110 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:14.454928 12110 tablet_bootstrap.cc:492] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:02:14.457779 12110 ts_tablet_manager.cc:1367] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.151s user 0.047s sys 0.012s
I20240501 14:02:14.458804 12110 raft_consensus.cc:348] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:14.459081 12110 raft_consensus.cc:729] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.459374 12110 consensus_queue.cc:260] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:14.460055 12110 ts_tablet_manager.cc:1398] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:14.460203 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(21c02994dfef4a10a87b1f47d3ab6e3b) complete. Timing: real 0.418s user 0.047s sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":31854,"dirs.queue_time_us":13811,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":18986,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":24274,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72283,"threads_started":3,"wal-append.queue_time_us":23412}
I20240501 14:02:14.464659 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(21c02994dfef4a10a87b1f47d3ab6e3b): 4921 bytes on disk
I20240501 14:02:14.469012 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(21c02994dfef4a10a87b1f47d3ab6e3b) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.473906 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-c2c63489706e4d068adde1872f451ad2 received from {username='slave'} at 127.22.33.72:49865
I20240501 14:02:14.474025 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-c2c63489706e4d068adde1872f451ad2 on tablet c2c63489706e4d068adde1872f451ad2 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:14.510959 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 9c7cd18adcd844f4be7ddd343cda3df4 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57709: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-9c7cd18adcd844f4be7ddd343cda3df4
I20240501 14:02:14.537926 12128 tablet_copy_client.cc:308] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:02:14.539397 32117 tablet_copy_source_session.cc:205] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:14.541006 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7cd18adcd844f4be7ddd343cda3df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.545295 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet ba3bd81448244b5bbdcd9eb8d23aee03 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:45945)
I20240501 14:02:14.545998 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet ba3bd81448244b5bbdcd9eb8d23aee03 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:45945: session id = 522f4ff5e9084cbf8462196533c7d7aa-ba3bd81448244b5bbdcd9eb8d23aee03
I20240501 14:02:14.548337 12123 tablet_copy_client.cc:783] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:14.548583 12123 tablet_copy_client.cc:647] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:14.548781 31985 tablet_copy_source_session.cc:205] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:14.551440 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3bd81448244b5bbdcd9eb8d23aee03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.552799 12123 tablet_copy_client.cc:523] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:14.557598 12128 tablet_copy_client.cc:783] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:14.557868 12128 tablet_copy_client.cc:647] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:14.566499 12128 tablet_copy_client.cc:523] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:14.571720 12123 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:14.574307 12128 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:14.611003 12145 ts_tablet_manager.cc:902] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:14.617990 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(21c02994dfef4a10a87b1f47d3ab6e3b) complete. Timing: real 0.391s user 0.067s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":6138,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2213,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3002,"threads_started":1,"wal-append.queue_time_us":410}
I20240501 14:02:14.620472 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(21c02994dfef4a10a87b1f47d3ab6e3b): 4921 bytes on disk
I20240501 14:02:14.624506 12128 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:14.624855 12128 tablet_bootstrap.cc:492] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:14.625123 12128 ts_tablet_manager.cc:1367] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.007s
I20240501 14:02:14.625901 12128 raft_consensus.cc:348] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:14.626116 12128 raft_consensus.cc:729] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.626993 12128 consensus_queue.cc:260] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:14.627983 12128 ts_tablet_manager.cc:1398] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:14.629293 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-ba3bd81448244b5bbdcd9eb8d23aee03 received from {username='slave'} at 127.22.33.67:45945
I20240501 14:02:14.629426 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-ba3bd81448244b5bbdcd9eb8d23aee03 on tablet ba3bd81448244b5bbdcd9eb8d23aee03 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:14.633935 12123 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:14.633989 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(21c02994dfef4a10a87b1f47d3ab6e3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.642489 12123 tablet_bootstrap.cc:492] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:14.642818 12123 ts_tablet_manager.cc:1367] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.011s
I20240501 14:02:14.643538 12123 raft_consensus.cc:348] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:14.643822 12123 raft_consensus.cc:729] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.644085 12123 consensus_queue.cc:260] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:14.648857 12123 ts_tablet_manager.cc:1398] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:02:14.649638 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-9c7cd18adcd844f4be7ddd343cda3df4 received from {username='slave'} at 127.22.33.68:57709
I20240501 14:02:14.649793 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-9c7cd18adcd844f4be7ddd343cda3df4 on tablet 9c7cd18adcd844f4be7ddd343cda3df4 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:14.665324 12145 tablet_copy_client.cc:308] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:14.666004 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 0132590f8ed5423f983c53d9c19f95fe from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:47475)
I20240501 14:02:14.666214 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 0132590f8ed5423f983c53d9c19f95fe from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:47475: session id = 522f4ff5e9084cbf8462196533c7d7aa-0132590f8ed5423f983c53d9c19f95fe
I20240501 14:02:14.668884 32117 tablet_copy_source_session.cc:205] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:14.673728 12145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0132590f8ed5423f983c53d9c19f95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.681385 12145 tablet_copy_client.cc:783] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:14.681602 12145 tablet_copy_client.cc:647] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:14.682934 12145 tablet_copy_client.cc:523] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:14.692859 12145 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:14.712869 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(8427dee24ad04247b23b24570c73c92a) complete. Timing: real 0.343s user 0.064s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1157,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7317,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35957,"threads_started":1,"wal-append.queue_time_us":35813}
I20240501 14:02:14.716800 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(8427dee24ad04247b23b24570c73c92a): 4581 bytes on disk
I20240501 14:02:14.717635 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(8427dee24ad04247b23b24570c73c92a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 14:02:14.735096 12145 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:14.735544 12145 tablet_bootstrap.cc:492] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:14.735808 12145 ts_tablet_manager.cc:1367] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.009s
I20240501 14:02:14.736507 12145 raft_consensus.cc:348] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:14.736724 12145 raft_consensus.cc:729] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.736986 12145 consensus_queue.cc:260] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:14.737679 12145 ts_tablet_manager.cc:1398] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:14.754575 32361 raft_consensus.cc:1203] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:14.782727 29635 raft_consensus.cc:1203] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:14.785112 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-0132590f8ed5423f983c53d9c19f95fe received from {username='slave'} at 127.22.33.67:47475
I20240501 14:02:14.785255 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-0132590f8ed5423f983c53d9c19f95fe on tablet 0132590f8ed5423f983c53d9c19f95fe with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:14.954811 29507 raft_consensus.cc:1203] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:15.142545 29507 raft_consensus.cc:1203] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:15.564254 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:15.574580 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:15.652765 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:15.672713 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:15.711069 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:15.741688 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:15.743081 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:15.765250 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:16.053990 32097 consensus_queue.cc:237] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:16.057907 32222 raft_consensus.cc:1261] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:16.058480 29506 raft_consensus.cc:1261] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:16.058449 12219 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:16.061235 12157 consensus_queue.cc:1035] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:16.064532 12219 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.068360 29506 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.068593 32224 raft_consensus.cc:2932] T 0132590f8ed5423f983c53d9c19f95fe P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.074203 29103 catalog_manager.cc:5484] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.083266 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0132590f8ed5423f983c53d9c19f95fe with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:16.084017 12248 tablet_replica.cc:330] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:16.084215 12248 raft_consensus.cc:2229] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:16.084441 12248 raft_consensus.cc:2258] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:16.085073 12248 ts_tablet_manager.cc:1875] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:16.090798 12248 ts_tablet_manager.cc:1888] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:16.090924 12248 log.cc:1196] T 0132590f8ed5423f983c53d9c19f95fe P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/0132590f8ed5423f983c53d9c19f95fe
I20240501 14:02:16.092219 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 0132590f8ed5423f983c53d9c19f95fe (table rebalancer_extra_table_32 [id=f5ad6b6ed31446419e08cccf2f35020e]) successfully deleted
I20240501 14:02:16.150184 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(0efddc01769f43e0b6f44d40ce6fa562): perf score=0.033342
I20240501 14:02:16.238847 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(0efddc01769f43e0b6f44d40ce6fa562): perf score=0.033367
I20240501 14:02:16.243837 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(9d39eab213cc464ba90149276cfeeac8): perf score=0.033371
I20240501 14:02:16.288656 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(0efddc01769f43e0b6f44d40ce6fa562) complete. Timing: real 0.138s user 0.065s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12613,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2332,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21652,"threads_started":2,"wal-append.queue_time_us":9027}
I20240501 14:02:16.291818 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(67ed6fab7d9e46aa8faeb3cf784af0fb): perf score=0.033380
I20240501 14:02:16.298123 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(9d39eab213cc464ba90149276cfeeac8): perf score=0.033386
I20240501 14:02:16.345395 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(9d39eab213cc464ba90149276cfeeac8) complete. Timing: real 0.101s user 0.051s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21121,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":6847,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":970,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27271,"threads_started":2,"wal-append.queue_time_us":5884}
I20240501 14:02:16.349409 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(9d39eab213cc464ba90149276cfeeac8): 4581 bytes on disk
I20240501 14:02:16.350179 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(9d39eab213cc464ba90149276cfeeac8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.386142 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(0efddc01769f43e0b6f44d40ce6fa562) complete. Timing: real 0.147s user 0.056s sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":21590,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":3432,"drs_written":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1084,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":36684,"threads_started":2,"wal-append.queue_time_us":10558}
I20240501 14:02:16.389576 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(0efddc01769f43e0b6f44d40ce6fa562): 4834 bytes on disk
I20240501 14:02:16.390527 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(0efddc01769f43e0b6f44d40ce6fa562) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.393230 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(67ed6fab7d9e46aa8faeb3cf784af0fb): perf score=0.033408
I20240501 14:02:16.393321 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(67ed6fab7d9e46aa8faeb3cf784af0fb) complete. Timing: real 0.098s user 0.050s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1569,"drs_written":1,"lbm_read_time_us":435,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8578,"threads_started":1,"wal-append.queue_time_us":8457}
I20240501 14:02:16.445430 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(9d39eab213cc464ba90149276cfeeac8) complete. Timing: real 0.147s user 0.074s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":64898,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1314,"drs_written":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":69856,"threads_started":2,"wal-append.queue_time_us":3306}
I20240501 14:02:16.449597 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(9d39eab213cc464ba90149276cfeeac8): 4581 bytes on disk
I20240501 14:02:16.451356 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(9d39eab213cc464ba90149276cfeeac8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.400940 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(67ed6fab7d9e46aa8faeb3cf784af0fb): 4922 bytes on disk
I20240501 14:02:16.489737 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(67ed6fab7d9e46aa8faeb3cf784af0fb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.489835 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(67ed6fab7d9e46aa8faeb3cf784af0fb) complete. Timing: real 0.096s user 0.055s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2989,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8418,"threads_started":1,"wal-append.queue_time_us":8313}
I20240501 14:02:16.492599 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(67ed6fab7d9e46aa8faeb3cf784af0fb): 4922 bytes on disk
I20240501 14:02:16.493686 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(67ed6fab7d9e46aa8faeb3cf784af0fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.511332 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(0efddc01769f43e0b6f44d40ce6fa562): 4834 bytes on disk
I20240501 14:02:16.512401 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(0efddc01769f43e0b6f44d40ce6fa562) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 14:02:16.551115 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:16.610026 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:16.635146 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:16.668190 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:16.731251 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:16.733803 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:16.734084 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:16.749873 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:16.946280 31963 consensus_queue.cc:237] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:16.948614 29506 raft_consensus.cc:1261] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:16.948959 32224 raft_consensus.cc:1261] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:16.949491 12304 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:16.949847 12075 consensus_queue.cc:1035] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:16.953685 12304 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.955018 32224 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.961542 29105 catalog_manager.cc:5484] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.965366 29506 raft_consensus.cc:2932] T ba3bd81448244b5bbdcd9eb8d23aee03 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:16.987903 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3bd81448244b5bbdcd9eb8d23aee03 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:02:16.988631 12353 tablet_replica.cc:330] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:02:16.988910 12353 raft_consensus.cc:2229] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:16.989210 12353 raft_consensus.cc:2258] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:16.990603 12353 ts_tablet_manager.cc:1875] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:16.996179 12353 ts_tablet_manager.cc:1888] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:16.996368 12353 log.cc:1196] T ba3bd81448244b5bbdcd9eb8d23aee03 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/ba3bd81448244b5bbdcd9eb8d23aee03
I20240501 14:02:16.997674 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet ba3bd81448244b5bbdcd9eb8d23aee03 (table rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]) successfully deleted
I20240501 14:02:17.331189 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:17.368636 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:17.385973 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:17.403146 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:17.409575 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:17.413305 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:17.435838 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:17.495956 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:02:17.553946 32163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 50341.1us
I20240501 14:02:17.668640 32097 consensus_queue.cc:237] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:17.670642 29634 raft_consensus.cc:1261] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:17.670673 32361 raft_consensus.cc:1261] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:17.671142 12157 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:17.671377 11597 consensus_queue.cc:1035] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:17.674177 12436 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:17.676098 29634 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:17.676831 32361 raft_consensus.cc:2932] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:17.682701 29101 catalog_manager.cc:5484] T 9c7cd18adcd844f4be7ddd343cda3df4 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:17.690706 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7cd18adcd844f4be7ddd343cda3df4 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41216
I20240501 14:02:17.692214 12441 tablet_replica.cc:330] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:02:17.692620 12441 raft_consensus.cc:2229] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:17.692831 12441 raft_consensus.cc:2258] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:17.693480 12441 ts_tablet_manager.cc:1875] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:17.699256 12441 ts_tablet_manager.cc:1888] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:17.699357 12441 log.cc:1196] T 9c7cd18adcd844f4be7ddd343cda3df4 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c7cd18adcd844f4be7ddd343cda3df4
I20240501 14:02:17.700089 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 9c7cd18adcd844f4be7ddd343cda3df4 (table rebalancer_extra_table_52 [id=9d03642db77f47678c8ba46b8f46c89d]) successfully deleted
I20240501 14:02:18.042481 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:18.059651 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:18.088387 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:18.105501 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:02:18.126884 32294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 53416.9us
I20240501 14:02:18.136480 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:18.154263 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:18.230106 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:18.236315 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:18.411691 29372 consensus_queue.cc:237] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:18.413920 32359 raft_consensus.cc:1261] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:18.413980 29246 raft_consensus.cc:1261] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:18.414433 11997 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:18.414752 12390 consensus_queue.cc:1035] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:18.417388 12390 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:18.417836 29246 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:18.428181 29105 catalog_manager.cc:5484] T c2c63489706e4d068adde1872f451ad2 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:18.433368 32359 raft_consensus.cc:2932] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:18.452121 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c63489706e4d068adde1872f451ad2 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:02:18.454365 12545 tablet_replica.cc:330] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:02:18.454648 12545 raft_consensus.cc:2229] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:18.454955 12545 raft_consensus.cc:2258] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:18.455734 12545 ts_tablet_manager.cc:1875] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:18.466228 12545 ts_tablet_manager.cc:1888] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:18.466435 12545 log.cc:1196] T c2c63489706e4d068adde1872f451ad2 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2c63489706e4d068adde1872f451ad2
I20240501 14:02:18.467890 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet c2c63489706e4d068adde1872f451ad2 (table rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]) successfully deleted
I20240501 14:02:18.781656 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:18.795677 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:18.799758 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:18.818024 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:18.821173 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:18.852821 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:18.887850 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:18.915297 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:19.098165 29634 consensus_queue.cc:237] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.100003 29507 raft_consensus.cc:1261] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:19.100044 32223 raft_consensus.cc:1261] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:19.100486 11521 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:19.100744 12412 consensus_queue.cc:1035] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:19.103585 12412 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.104065 32223 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.104034 29507 raft_consensus.cc:2932] T 8a1690e3a76644859a0711e305b8f946 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.111320 29100 catalog_manager.cc:5484] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.127631 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a1690e3a76644859a0711e305b8f946 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:19.133407 12625 tablet_replica.cc:330] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:19.133635 12625 raft_consensus.cc:2229] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:19.133864 12625 raft_consensus.cc:2258] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:19.134718 12625 ts_tablet_manager.cc:1875] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:19.141597 12625 ts_tablet_manager.cc:1888] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:19.141870 12625 log.cc:1196] T 8a1690e3a76644859a0711e305b8f946 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/8a1690e3a76644859a0711e305b8f946
I20240501 14:02:19.142882 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 8a1690e3a76644859a0711e305b8f946 (table rebalancer_extra_table_8 [id=c4774d7871124ad69e2971b30257f095]) successfully deleted
I20240501 14:02:19.294132 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(a87b1de51c4c4f909c27c27aa745fb44): perf score=0.033343
I20240501 14:02:19.339032 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(a87b1de51c4c4f909c27c27aa745fb44) complete. Timing: real 0.045s user 0.032s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1963,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1542,"threads_started":2,"wal-append.queue_time_us":1409}
I20240501 14:02:19.340956 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(d2e5851626b54e1caf6105a70ccafc43): perf score=0.033352
I20240501 14:02:19.379213 32223 consensus_queue.cc:237] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.382036 32095 raft_consensus.cc:1261] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:19.382694 12415 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:19.382952 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(d2e5851626b54e1caf6105a70ccafc43) complete. Timing: real 0.042s user 0.028s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":3414,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":249,"threads_started":1,"wal-append.queue_time_us":373}
I20240501 14:02:19.383435 32361 raft_consensus.cc:1261] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:19.385488 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(a87b1de51c4c4f909c27c27aa745fb44): 4921 bytes on disk
W20240501 14:02:19.385916 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad7af304700545b2a28249f9f046cab6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:19.386189 12415 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:19.386446 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(d2e5851626b54e1caf6105a70ccafc43): perf score=0.033354
I20240501 14:02:19.393884 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(a87b1de51c4c4f909c27c27aa745fb44) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.397322 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(d2e5851626b54e1caf6105a70ccafc43): 4581 bytes on disk
I20240501 14:02:19.397936 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(d2e5851626b54e1caf6105a70ccafc43) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.406178 12689 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.406775 32097 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.413038 29105 catalog_manager.cc:5484] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.413197 32361 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.449402 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(d2e5851626b54e1caf6105a70ccafc43) complete. Timing: real 0.063s user 0.028s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5855,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":4360,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7127,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8300,"threads_started":2,"wal-append.queue_time_us":2645}
I20240501 14:02:19.452064 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(d2e5851626b54e1caf6105a70ccafc43): 4581 bytes on disk
I20240501 14:02:19.452781 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(d2e5851626b54e1caf6105a70ccafc43) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.487040 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(c6973301bdd94c2b8f98e718715ff752): perf score=0.033380
I20240501 14:02:19.531298 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(a87b1de51c4c4f909c27c27aa745fb44): perf score=0.033401
W20240501 14:02:19.563795 32031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 61847.2us
I20240501 14:02:19.608171 32361 consensus_queue.cc:237] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.611848 29246 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:19.613763 12413 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:02:19.614228 32295 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10e1775ccab942ff857d22e89d99764d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:19.616712 29635 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:19.617475 12413 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:19.624894 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(c6973301bdd94c2b8f98e718715ff752) complete. Timing: real 0.137s user 0.042s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1481,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1030,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":553,"threads_started":2,"wal-append.queue_time_us":309}
I20240501 14:02:19.626660 12512 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.627339 29246 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.631085 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752): 4834 bytes on disk
I20240501 14:02:19.631599 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.634315 29635 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.634555 29105 catalog_manager.cc:5484] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.635963 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(a87b1de51c4c4f909c27c27aa745fb44) complete. Timing: real 0.104s user 0.046s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":966,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1016,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4650,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1103,"threads_started":2,"wal-append.queue_time_us":1185}
I20240501 14:02:19.639251 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(a87b1de51c4c4f909c27c27aa745fb44): 4921 bytes on disk
I20240501 14:02:19.639865 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(a87b1de51c4c4f909c27c27aa745fb44) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.643288 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(c6973301bdd94c2b8f98e718715ff752): perf score=0.033417
I20240501 14:02:19.690178 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(c6973301bdd94c2b8f98e718715ff752) complete. Timing: real 0.047s user 0.024s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":864,"threads_started":1,"wal-append.queue_time_us":956}
I20240501 14:02:19.693105 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752): 4834 bytes on disk
I20240501 14:02:19.693715 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240501 14:02:19.696385 29246 consensus_queue.cc:237] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:19.699084 29635 raft_consensus.cc:1261] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:19.699234 29181 consensus_peers.cc:485] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83ca18280bbb49058e5aa80a3832b666. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:19.699352 29507 raft_consensus.cc:1261] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:19.699601 12678 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:19.699862 12516 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:19.702797 12678 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.703776 29507 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.708506 29635 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.710680 29100 catalog_manager.cc:5484] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.745769 31963 consensus_queue.cc:237] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.748556 32361 raft_consensus.cc:1261] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:19.749172 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8020272a8b14dea8a6a82802ea4e10a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:19.749503 12075 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:19.750977 29246 raft_consensus.cc:1261] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:19.751510 12075 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:19.753940 12075 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.754521 29246 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.760644 29100 catalog_manager.cc:5484] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.770543 32361 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:19.797037 29372 consensus_queue.cc:237] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
W20240501 14:02:19.799355 29312 consensus_peers.cc:485] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e33e4f9b1204797aeff0bca36c93911. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:19.799559 29507 raft_consensus.cc:1261] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:19.799774 29635 raft_consensus.cc:1261] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:19.800051 12536 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:19.800333 11997 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:19.803547 12536 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.804324 29635 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.808694 29507 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.821453 29105 catalog_manager.cc:5484] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index -1 to 4, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:19.912662 12790 ts_tablet_manager.cc:902] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:19.930968 12790 tablet_copy_client.cc:308] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.71:38743
I20240501 14:02:19.931824 32248 tablet_copy_service.cc:140] P 6a446ac802a7457d9f27076781f21d79: Received BeginTabletCopySession request for tablet ad7af304700545b2a28249f9f046cab6 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:38119)
I20240501 14:02:19.931988 32248 tablet_copy_service.cc:161] P 6a446ac802a7457d9f27076781f21d79: Beginning new tablet copy session on tablet ad7af304700545b2a28249f9f046cab6 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:38119: session id = 522f4ff5e9084cbf8462196533c7d7aa-ad7af304700545b2a28249f9f046cab6
I20240501 14:02:19.935972 32248 tablet_copy_source_session.cc:205] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:19.937393 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7af304700545b2a28249f9f046cab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.944525 12790 tablet_copy_client.cc:783] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:19.944952 12790 tablet_copy_client.cc:647] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:19.946858 12790 tablet_copy_client.cc:523] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:19.950724 12790 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:19.965759 12790 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:19.966287 12790 tablet_bootstrap.cc:492] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:19.966619 12790 ts_tablet_manager.cc:1367] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240501 14:02:19.967361 12790 raft_consensus.cc:348] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.967651 12790 raft_consensus.cc:729] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.967928 12790 consensus_queue.cc:260] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:19.968451 12790 ts_tablet_manager.cc:1398] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:19.969412 32248 tablet_copy_service.cc:342] P 6a446ac802a7457d9f27076781f21d79: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-ad7af304700545b2a28249f9f046cab6 received from {username='slave'} at 127.22.33.67:38119
I20240501 14:02:19.969523 32248 tablet_copy_service.cc:433] P 6a446ac802a7457d9f27076781f21d79: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-ad7af304700545b2a28249f9f046cab6 on tablet ad7af304700545b2a28249f9f046cab6 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:20.029363 12790 ts_tablet_manager.cc:902] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:20.030579 12790 tablet_copy_client.cc:308] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:02:20.032490 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet 10e1775ccab942ff857d22e89d99764d from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:51381)
I20240501 14:02:20.032646 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet 10e1775ccab942ff857d22e89d99764d from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:51381: session id = 522f4ff5e9084cbf8462196533c7d7aa-10e1775ccab942ff857d22e89d99764d
I20240501 14:02:20.068219 32383 tablet_copy_source_session.cc:205] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:20.069586 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e1775ccab942ff857d22e89d99764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.085284 12790 tablet_copy_client.cc:783] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:20.085645 12790 tablet_copy_client.cc:647] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:20.089545 12790 tablet_copy_client.cc:523] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:20.093124 12790 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:20.114094 12790 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:20.114892 12790 tablet_bootstrap.cc:492] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:20.115336 12790 ts_tablet_manager.cc:1367] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.009s
I20240501 14:02:20.117152 12790 raft_consensus.cc:348] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:20.117547 12790 raft_consensus.cc:729] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.118057 12790 consensus_queue.cc:260] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:20.118914 12790 ts_tablet_manager.cc:1398] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:20.119827 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-10e1775ccab942ff857d22e89d99764d received from {username='slave'} at 127.22.33.67:51381
I20240501 14:02:20.119954 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-10e1775ccab942ff857d22e89d99764d on tablet 10e1775ccab942ff857d22e89d99764d with peer 522f4ff5e9084cbf8462196533c7d7aa
W20240501 14:02:20.205384 29441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 45053.8us
I20240501 14:02:20.279969 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:20.281339 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:20.287891 12848 ts_tablet_manager.cc:902] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:20.301402 12847 ts_tablet_manager.cc:902] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:20.329815 12790 ts_tablet_manager.cc:902] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:20.337106 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:20.346491 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:20.398068 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:20.409099 12847 tablet_copy_client.cc:308] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:20.410702 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 0e33e4f9b1204797aeff0bca36c93911 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:38381)
I20240501 14:02:20.410920 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 0e33e4f9b1204797aeff0bca36c93911 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:38381: session id = 0a697f3a1ee544daa7073814ab845d58-0e33e4f9b1204797aeff0bca36c93911
I20240501 14:02:20.414809 29397 tablet_copy_source_session.cc:205] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:20.416731 12847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e33e4f9b1204797aeff0bca36c93911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.447175 12847 tablet_copy_client.cc:783] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:20.447873 12847 tablet_copy_client.cc:647] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:20.468502 29507 raft_consensus.cc:1203] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:20.488443 29507 raft_consensus.cc:1203] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:20.525892 12848 tablet_copy_client.cc:308] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:02:20.526707 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 83ca18280bbb49058e5aa80a3832b666 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36699)
I20240501 14:02:20.526906 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 83ca18280bbb49058e5aa80a3832b666 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36699: session id = 0a697f3a1ee544daa7073814ab845d58-83ca18280bbb49058e5aa80a3832b666
I20240501 14:02:20.530369 29266 tablet_copy_source_session.cc:205] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:20.533589 12848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ca18280bbb49058e5aa80a3832b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.541946 12848 tablet_copy_client.cc:783] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:20.542373 12848 tablet_copy_client.cc:647] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:20.542562 12847 tablet_copy_client.cc:523] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:20.544682 12848 tablet_copy_client.cc:523] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:20.546348 12847 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:02:20.548578 12848 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
I20240501 14:02:20.602025 12790 tablet_copy_client.cc:308] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:02:20.603435 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet c8020272a8b14dea8a6a82802ea4e10a from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:45945)
I20240501 14:02:20.603600 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet c8020272a8b14dea8a6a82802ea4e10a from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:45945: session id = 522f4ff5e9084cbf8462196533c7d7aa-c8020272a8b14dea8a6a82802ea4e10a
I20240501 14:02:20.606607 31985 tablet_copy_source_session.cc:205] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:20.606891 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:20.625370 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8020272a8b14dea8a6a82802ea4e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.646301 12790 tablet_copy_client.cc:783] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:20.646649 12790 tablet_copy_client.cc:647] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:20.646797 12848 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:20.647253 12848 tablet_bootstrap.cc:492] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:02:20.647541 12848 ts_tablet_manager.cc:1367] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.099s user 0.056s sys 0.004s
I20240501 14:02:20.648275 12848 raft_consensus.cc:348] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:20.648509 12848 raft_consensus.cc:729] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.648772 12848 consensus_queue.cc:260] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:20.649677 12848 ts_tablet_manager.cc:1398] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:20.650321 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-83ca18280bbb49058e5aa80a3832b666 received from {username='slave'} at 127.22.33.70:36699
I20240501 14:02:20.650444 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-83ca18280bbb49058e5aa80a3832b666 on tablet 83ca18280bbb49058e5aa80a3832b666 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:02:20.650568 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:20.652905 12790 tablet_copy_client.cc:523] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:20.656868 12790 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:20.664324 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:20.699198 12790 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:20.699574 12790 tablet_bootstrap.cc:492] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:20.699891 12790 ts_tablet_manager.cc:1367] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.004s
I20240501 14:02:20.700585 12790 raft_consensus.cc:348] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:20.701045 12790 raft_consensus.cc:729] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.701442 12790 consensus_queue.cc:260] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:20.705325 12790 ts_tablet_manager.cc:1398] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:20.705991 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-c8020272a8b14dea8a6a82802ea4e10a received from {username='slave'} at 127.22.33.67:45945
I20240501 14:02:20.706115 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-c8020272a8b14dea8a6a82802ea4e10a on tablet c8020272a8b14dea8a6a82802ea4e10a with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:20.735608 32095 raft_consensus.cc:1203] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:20.792300 12847 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:20.793503 12847 tablet_bootstrap.cc:492] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:02:20.793802 12847 ts_tablet_manager.cc:1367] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.248s user 0.047s sys 0.002s
I20240501 14:02:20.795338 12847 raft_consensus.cc:348] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:20.795586 12847 raft_consensus.cc:729] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.795943 12847 consensus_queue.cc:260] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:20.798391 12847 ts_tablet_manager.cc:1398] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:20.800280 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-0e33e4f9b1204797aeff0bca36c93911 received from {username='slave'} at 127.22.33.70:38381
I20240501 14:02:20.800410 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-0e33e4f9b1204797aeff0bca36c93911 on tablet 0e33e4f9b1204797aeff0bca36c93911 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:02:20.899199 29507 raft_consensus.cc:1203] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:21.127337 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:21.133646 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:21.148015 32095 raft_consensus.cc:1203] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:21.171497 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:21.183478 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:21.230430 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:21.294003 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:21.296200 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:21.308499 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:21.407284 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(ca0a888dc37d44c69dea73f9ffd556f6): perf score=0.033372
I20240501 14:02:21.435406 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(f345c7a43d2648b4b4c47e726c4d4c51): perf score=0.033377
I20240501 14:02:21.472214 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(ca0a888dc37d44c69dea73f9ffd556f6): perf score=0.033391
I20240501 14:02:21.479228 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(3d8ce8f6f27b4ab19ff5a1bb972ca202): perf score=0.033391
I20240501 14:02:21.523669 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(ca0a888dc37d44c69dea73f9ffd556f6) complete. Timing: real 0.116s user 0.054s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12740,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1961,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":832,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18582,"threads_started":2,"wal-append.queue_time_us":6778}
I20240501 14:02:21.527933 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(ca0a888dc37d44c69dea73f9ffd556f6): 4921 bytes on disk
I20240501 14:02:21.538278 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(ca0a888dc37d44c69dea73f9ffd556f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.541393 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(3d8ce8f6f27b4ab19ff5a1bb972ca202): perf score=0.033405
I20240501 14:02:21.593891 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(ca0a888dc37d44c69dea73f9ffd556f6) complete. Timing: real 0.121s user 0.038s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6322,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":10174,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14218,"threads_started":2,"wal-append.queue_time_us":3353}
I20240501 14:02:21.642177 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(ca0a888dc37d44c69dea73f9ffd556f6): 4921 bytes on disk
I20240501 14:02:21.643365 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(ca0a888dc37d44c69dea73f9ffd556f6) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.644742 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(3d8ce8f6f27b4ab19ff5a1bb972ca202) complete. Timing: real 0.103s user 0.043s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6247,"threads_started":1,"wal-append.queue_time_us":6348}
I20240501 14:02:21.647657 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(3d8ce8f6f27b4ab19ff5a1bb972ca202): 4581 bytes on disk
I20240501 14:02:21.652959 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(f345c7a43d2648b4b4c47e726c4d4c51): perf score=0.033435
I20240501 14:02:21.661648 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(3d8ce8f6f27b4ab19ff5a1bb972ca202) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.675653 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(f345c7a43d2648b4b4c47e726c4d4c51) complete. Timing: real 0.234s user 0.075s sys 0.072s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":20570,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":7627,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1032,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127279,"threads_started":2,"wal-append.queue_time_us":106654}
I20240501 14:02:21.679132 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(f345c7a43d2648b4b4c47e726c4d4c51): 4834 bytes on disk
I20240501 14:02:21.681493 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(f345c7a43d2648b4b4c47e726c4d4c51) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.684135 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(3d8ce8f6f27b4ab19ff5a1bb972ca202) complete. Timing: real 0.205s user 0.036s sys 0.041s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12685,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":8721,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58278,"threads_started":2,"wal-append.queue_time_us":44474}
I20240501 14:02:21.688324 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(3d8ce8f6f27b4ab19ff5a1bb972ca202): 4581 bytes on disk
I20240501 14:02:21.689256 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(3d8ce8f6f27b4ab19ff5a1bb972ca202) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.708093 31963 consensus_queue.cc:237] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:21.710255 29507 raft_consensus.cc:1261] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:21.710436 29244 raft_consensus.cc:1261] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:21.711058 12935 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:21.711354 12871 consensus_queue.cc:1035] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:21.714563 12935 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:21.715330 29507 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:21.718695 29245 raft_consensus.cc:2932] T c8020272a8b14dea8a6a82802ea4e10a P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:21.725142 29104 catalog_manager.cc:5484] T c8020272a8b14dea8a6a82802ea4e10a P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 2 to 3, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:21.734746 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(f345c7a43d2648b4b4c47e726c4d4c51) complete. Timing: real 0.074s user 0.031s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":4480,"thread_start_us":243,"threads_started":1,"wal-append.queue_time_us":385}
I20240501 14:02:21.739589 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(f345c7a43d2648b4b4c47e726c4d4c51): 4834 bytes on disk
I20240501 14:02:21.742201 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(f345c7a43d2648b4b4c47e726c4d4c51) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":203,"lbm_reads_lt_1ms":4}
I20240501 14:02:21.747799 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet c8020272a8b14dea8a6a82802ea4e10a with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:21.748368 12980 tablet_replica.cc:330] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:21.748688 12980 raft_consensus.cc:2229] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:21.749032 12980 raft_consensus.cc:2258] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:21.750072 12980 ts_tablet_manager.cc:1875] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:21.756145 12980 ts_tablet_manager.cc:1888] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:21.756326 12980 log.cc:1196] T c8020272a8b14dea8a6a82802ea4e10a P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/c8020272a8b14dea8a6a82802ea4e10a
I20240501 14:02:21.757068 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet c8020272a8b14dea8a6a82802ea4e10a (table rebalancer_extra_table_4 [id=f4e5f6d094234e1fac8d9875ac375ed9]) successfully deleted
I20240501 14:02:22.089040 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:22.120805 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:22.127087 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:22.141238 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:22.183862 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:22.199842 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:22.228312 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:22.293273 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:22.404161 29244 consensus_queue.cc:237] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:22.406121 32095 raft_consensus.cc:1261] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:22.406114 29634 raft_consensus.cc:1261] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:22.406641 12936 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:22.406929 12938 consensus_queue.cc:1035] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:22.410535 12938 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:22.411235 32095 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:22.417804 29634 raft_consensus.cc:2932] T 83ca18280bbb49058e5aa80a3832b666 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:22.420467 29104 catalog_manager.cc:5484] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:22.436543 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ca18280bbb49058e5aa80a3832b666 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:02:22.437729 13063 tablet_replica.cc:330] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:02:22.437942 13063 raft_consensus.cc:2229] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:22.438287 13063 raft_consensus.cc:2258] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:22.439100 13063 ts_tablet_manager.cc:1875] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:22.445662 13063 ts_tablet_manager.cc:1888] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:22.445875 13063 log.cc:1196] T 83ca18280bbb49058e5aa80a3832b666 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/83ca18280bbb49058e5aa80a3832b666
I20240501 14:02:22.447587 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 83ca18280bbb49058e5aa80a3832b666 (table rebalance_test_table_11 [id=e2b1173c6aad461d8f39f25a09d9d7a1]) successfully deleted
I20240501 14:02:22.766130 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:22.766878 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:22.786584 32342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:22.795277 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:22.800410 29485 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:22.813715 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:22.831301 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 14:02:22.986327 31900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 118728us
I20240501 14:02:23.014153 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:23.089053 32361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d"
dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53214
I20240501 14:02:23.089295 32361 raft_consensus.cc:595] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:23.217056 12955 raft_consensus.cc:979] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: : Instructing follower 522f4ff5e9084cbf8462196533c7d7aa to start an election
I20240501 14:02:23.217228 12955 raft_consensus.cc:1067] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Signalling peer 522f4ff5e9084cbf8462196533c7d7aa to start an election
I20240501 14:02:23.218252 29507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d"
dest_uuid: "522f4ff5e9084cbf8462196533c7d7aa"
from {username='slave'} at 127.22.33.72:44367
I20240501 14:02:23.218468 29507 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:23.218583 29507 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:23.223385 29507 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:23.224680 29244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:02:23.224856 29244 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:23.227524 29244 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 2.
I20240501 14:02:23.228710 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:02:23.228919 32361 raft_consensus.cc:3032] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:23.229310 32361 raft_consensus.cc:729] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:02:23.229586 32361 consensus_queue.cc:260] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:23.229957 32361 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:23.232587 32361 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 2.
I20240501 14:02:23.233417 29441 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:02:23.233677 12416 raft_consensus.cc:2781] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:23.233772 12416 raft_consensus.cc:686] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 LEADER]: Becoming Leader. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Running, Role: LEADER
I20240501 14:02:23.233995 12416 consensus_queue.cc:237] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:23.239126 29507 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:23.239305 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "522f4ff5e9084cbf8462196533c7d7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:02:23.239512 29634 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:23.243953 29634 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522f4ff5e9084cbf8462196533c7d7aa in term 2.
I20240501 14:02:23.256208 29104 catalog_manager.cc:5484] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: term changed from 1 to 2, leader changed from 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) to 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67). New cstate: current_term: 2 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:23.409121 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:23.422180 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:23.456533 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:02:23.466025 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:23.478271 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:23.546187 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:23.569195 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:23.586361 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:23.634742 29245 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:02:23.635337 12416 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:23.688616 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(a6e94a79c4514fe4a3ae8e41c7792dc9): perf score=0.033377
I20240501 14:02:23.702253 29635 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:02:23.706187 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(49b72e2ac9df4426b071a7e5004e12e1): perf score=0.033377
I20240501 14:02:23.708956 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(77ce342cbfad4419b3e2e8d3a52c4628): perf score=0.033379
I20240501 14:02:23.718569 12416 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:23.776044 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(a6e94a79c4514fe4a3ae8e41c7792dc9): perf score=0.033400
I20240501 14:02:23.779815 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(a6e94a79c4514fe4a3ae8e41c7792dc9) complete. Timing: real 0.091s user 0.034s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3338,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1738,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3435,"threads_started":2,"wal-append.queue_time_us":318}
I20240501 14:02:23.782681 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(a6e94a79c4514fe4a3ae8e41c7792dc9): 4834 bytes on disk
I20240501 14:02:23.783247 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(a6e94a79c4514fe4a3ae8e41c7792dc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 14:02:23.801669 32361 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:02:23.802291 12416 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:23.937192 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(77ce342cbfad4419b3e2e8d3a52c4628) complete. Timing: real 0.228s user 0.084s sys 0.047s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":109458,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":8756,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140148,"threads_started":2,"wal-append.queue_time_us":15682}
I20240501 14:02:23.940233 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(49b72e2ac9df4426b071a7e5004e12e1): perf score=0.033439
I20240501 14:02:23.950556 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(49b72e2ac9df4426b071a7e5004e12e1) complete. Timing: real 0.243s user 0.044s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14870,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":2826,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1832,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":24405,"threads_started":2,"wal-append.queue_time_us":9546}
I20240501 14:02:23.953462 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(49b72e2ac9df4426b071a7e5004e12e1): 4921 bytes on disk
I20240501 14:02:23.954224 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(49b72e2ac9df4426b071a7e5004e12e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 14:02:23.967473 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(a6e94a79c4514fe4a3ae8e41c7792dc9) complete. Timing: real 0.191s user 0.045s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15790,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":6165,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":995,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16168,"threads_started":2,"wal-append.queue_time_us":335}
I20240501 14:02:23.970280 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(a6e94a79c4514fe4a3ae8e41c7792dc9): 4834 bytes on disk
I20240501 14:02:23.972332 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(a6e94a79c4514fe4a3ae8e41c7792dc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240501 14:02:23.976542 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(77ce342cbfad4419b3e2e8d3a52c4628): perf score=0.033453
I20240501 14:02:24.032313 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(49b72e2ac9df4426b071a7e5004e12e1) complete. Timing: real 0.091s user 0.054s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":354,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":4739,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":24827,"threads_started":1,"wal-append.queue_time_us":24920}
I20240501 14:02:24.035634 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(49b72e2ac9df4426b071a7e5004e12e1): 4921 bytes on disk
I20240501 14:02:24.039312 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(49b72e2ac9df4426b071a7e5004e12e1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4}
I20240501 14:02:24.043148 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(77ce342cbfad4419b3e2e8d3a52c4628): 4581 bytes on disk
I20240501 14:02:24.043715 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(77ce342cbfad4419b3e2e8d3a52c4628) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 14:02:24.051337 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(77ce342cbfad4419b3e2e8d3a52c4628) complete. Timing: real 0.074s user 0.044s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1091,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7279,"threads_started":1,"wal-append.queue_time_us":7173}
I20240501 14:02:24.054351 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(77ce342cbfad4419b3e2e8d3a52c4628): 4581 bytes on disk
I20240501 14:02:24.055083 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(77ce342cbfad4419b3e2e8d3a52c4628) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:02:24.253872 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:24.382002 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:02:24.388298 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:24.442246 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:24.456653 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:24.473423 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:24.482583 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:24.520994 32163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 48998.1us
I20240501 14:02:24.528147 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:24.681193 32224 consensus_queue.cc:237] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:24.683446 29507 raft_consensus.cc:1261] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:24.683604 32361 raft_consensus.cc:1261] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:24.683954 13033 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:24.684314 13282 consensus_queue.cc:1035] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:24.687104 13282 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:24.688145 29507 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:24.688192 32361 raft_consensus.cc:2932] T ad7af304700545b2a28249f9f046cab6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:24.694936 29103 catalog_manager.cc:5484] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 2 to 3, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) evicted. New cstate: current_term: 1 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:24.700716 32076 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7af304700545b2a28249f9f046cab6 with delete_type TABLET_DATA_TOMBSTONED (TS 0a697f3a1ee544daa7073814ab845d58 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33396
I20240501 14:02:24.701388 13306 tablet_replica.cc:330] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: stopping tablet replica
I20240501 14:02:24.701789 13306 raft_consensus.cc:2229] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:24.702159 13306 raft_consensus.cc:2258] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:24.702807 13306 ts_tablet_manager.cc:1875] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:24.708315 13306 ts_tablet_manager.cc:1888] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:24.708462 13306 log.cc:1196] T ad7af304700545b2a28249f9f046cab6 P 0a697f3a1ee544daa7073814ab845d58: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/wals/ad7af304700545b2a28249f9f046cab6
I20240501 14:02:24.709175 29091 catalog_manager.cc:4830] TS 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): tablet ad7af304700545b2a28249f9f046cab6 (table rebalancer_extra_table_6 [id=aa51025b883748b986a6deefb4b7cda6]) successfully deleted
I20240501 14:02:25.168040 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:25.228739 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:02:25.318688 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:25.363229 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:25.403815 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:25.406737 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:25.410547 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:25.497781 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:25.619946 29377 consensus_queue.cc:237] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:25.621999 29634 raft_consensus.cc:1261] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:25.622027 32096 raft_consensus.cc:1261] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:25.622589 12536 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:25.623466 12933 consensus_queue.cc:1035] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:25.625563 12536 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:25.626571 32096 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:25.632234 29097 catalog_manager.cc:5484] T 0e33e4f9b1204797aeff0bca36c93911 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:25.633452 29634 raft_consensus.cc:2932] T 0e33e4f9b1204797aeff0bca36c93911 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:25.655319 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e33e4f9b1204797aeff0bca36c93911 with delete_type TABLET_DATA_TOMBSTONED (TS 522f4ff5e9084cbf8462196533c7d7aa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41216
I20240501 14:02:25.657348 13392 tablet_replica.cc:330] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: stopping tablet replica
I20240501 14:02:25.657611 13392 raft_consensus.cc:2229] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:25.657871 13392 raft_consensus.cc:2258] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:25.658572 13392 ts_tablet_manager.cc:1875] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:25.664912 13392 ts_tablet_manager.cc:1888] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:25.665061 13392 log.cc:1196] T 0e33e4f9b1204797aeff0bca36c93911 P 522f4ff5e9084cbf8462196533c7d7aa: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e33e4f9b1204797aeff0bca36c93911
I20240501 14:02:25.666110 29090 catalog_manager.cc:4830] TS 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): tablet 0e33e4f9b1204797aeff0bca36c93911 (table rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]) successfully deleted
I20240501 14:02:25.831957 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(70ffd2100f104b339bfdc2d272ba3729): perf score=0.033333
I20240501 14:02:25.965329 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(70ffd2100f104b339bfdc2d272ba3729) complete. Timing: real 0.133s user 0.044s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49845,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":2260,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":53657,"threads_started":2,"wal-append.queue_time_us":7697}
I20240501 14:02:25.967761 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(70ffd2100f104b339bfdc2d272ba3729): 4921 bytes on disk
I20240501 14:02:25.968374 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(70ffd2100f104b339bfdc2d272ba3729) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:02:25.971040 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(241a2fdfe954476ba9bd2cfeb4b931ac): perf score=0.033365
I20240501 14:02:25.980409 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling FlushMRSOp(70ffd2100f104b339bfdc2d272ba3729): perf score=0.033368
I20240501 14:02:26.068879 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(83ca18280bbb49058e5aa80a3832b666): perf score=0.033396
I20240501 14:02:26.066229 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(83ca18280bbb49058e5aa80a3832b666): perf score=0.033397
I20240501 14:02:26.104195 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:02:26.122303 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:26.145592 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: FlushMRSOp(70ffd2100f104b339bfdc2d272ba3729) complete. Timing: real 0.165s user 0.064s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":20585,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29311,"threads_started":2,"wal-append.queue_time_us":8904}
I20240501 14:02:26.148677 29553 maintenance_manager.cc:419] P 522f4ff5e9084cbf8462196533c7d7aa: Scheduling UndoDeltaBlockGCOp(70ffd2100f104b339bfdc2d272ba3729): 4921 bytes on disk
I20240501 14:02:26.149303 29444 maintenance_manager.cc:643] P 522f4ff5e9084cbf8462196533c7d7aa: UndoDeltaBlockGCOp(70ffd2100f104b339bfdc2d272ba3729) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:02:26.171147 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(241a2fdfe954476ba9bd2cfeb4b931ac) complete. Timing: real 0.197s user 0.046s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":24320,"thread_start_us":5269,"threads_started":1,"wal-append.queue_time_us":5142}
I20240501 14:02:26.175042 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(241a2fdfe954476ba9bd2cfeb4b931ac): 4834 bytes on disk
I20240501 14:02:26.175894 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(241a2fdfe954476ba9bd2cfeb4b931ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 14:02:26.217445 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(83ca18280bbb49058e5aa80a3832b666) complete. Timing: real 0.122s user 0.043s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32916,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":10445,"drs_written":1,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4,"lbm_write_time_us":892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":39350,"threads_started":2,"wal-append.queue_time_us":5225}
I20240501 14:02:26.220909 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(83ca18280bbb49058e5aa80a3832b666): 4581 bytes on disk
I20240501 14:02:26.226490 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(83ca18280bbb49058e5aa80a3832b666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:02:26.238034 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(241a2fdfe954476ba9bd2cfeb4b931ac): perf score=0.033437
I20240501 14:02:26.254868 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:26.325037 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:26.374760 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:26.390515 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:26.412770 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:26.435295 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(83ca18280bbb49058e5aa80a3832b666) complete. Timing: real 0.366s user 0.038s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13341,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1513,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28058,"threads_started":2,"wal-append.queue_time_us":8835}
I20240501 14:02:26.447309 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(83ca18280bbb49058e5aa80a3832b666): 4581 bytes on disk
I20240501 14:02:26.447973 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(83ca18280bbb49058e5aa80a3832b666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 14:02:26.455011 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(241a2fdfe954476ba9bd2cfeb4b931ac) complete. Timing: real 0.217s user 0.051s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2666,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5449,"threads_started":1,"wal-append.queue_time_us":685}
I20240501 14:02:26.457612 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(241a2fdfe954476ba9bd2cfeb4b931ac): 4834 bytes on disk
I20240501 14:02:26.458190 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(241a2fdfe954476ba9bd2cfeb4b931ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240501 14:02:26.477587 29572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.66:39595, user_credentials={real_user=slave}} blocked reactor thread for 82849.9us
I20240501 14:02:26.496381 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:26.623919 29505 consensus_queue.cc:237] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:26.626195 29245 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:26.626225 29635 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:26.626691 12416 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:26.628373 12416 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:26.629477 13403 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P 522f4ff5e9084cbf8462196533c7d7aa [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:26.630033 29635 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:26.633715 29244 raft_consensus.cc:2932] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:26.637429 29100 catalog_manager.cc:5484] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: config changed from index 2 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 2 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:26.651746 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e1775ccab942ff857d22e89d99764d with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:26.662743 13508 tablet_replica.cc:330] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:26.662997 13508 raft_consensus.cc:2229] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:26.663214 13508 raft_consensus.cc:2258] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:26.663918 13508 ts_tablet_manager.cc:1875] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:26.669463 13508 ts_tablet_manager.cc:1888] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 14:02:26.669605 13508 log.cc:1196] T 10e1775ccab942ff857d22e89d99764d P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/10e1775ccab942ff857d22e89d99764d
I20240501 14:02:26.670260 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet 10e1775ccab942ff857d22e89d99764d (table rebalancer_extra_table_48 [id=7611916edbbf487a857aca2e0d8753df]) successfully deleted
I20240501 14:02:27.021742 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:27.023228 29224 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:27.045887 29356 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:27.046382 29485 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:02:27.057920 32207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:02:27.116185 29617 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:02:27.120402 31944 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:02:27.126004 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:27.399967 32360 consensus_queue.cc:237] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:27.402837 29245 raft_consensus.cc:1261] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:27.403144 29370 raft_consensus.cc:1261] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:27.403183 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ec202bb41a44e05939f1ecdf33d8c38. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.403517 13443 consensus_queue.cc:1035] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:27.404522 13443 consensus_queue.cc:1035] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:27.407187 13443 raft_consensus.cc:2932] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.407783 29370 raft_consensus.cc:2932] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.411147 29245 raft_consensus.cc:2932] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.418185 29100 catalog_manager.cc:5484] T 3ec202bb41a44e05939f1ecdf33d8c38 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.459249 29245 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:27.462522 32360 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:27.463941 29370 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:27.464366 13571 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:27.464653 12938 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:27.465143 29181 consensus_peers.cc:485] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 578c3f4c5a7c4e2c962b64736195f0bc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.481635 12938 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:27.486436 29374 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:27.489321 32360 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:27.492940 29099 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:27.581175 32097 consensus_queue.cc:237] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:27.586373 29506 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:27.586982 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.587383 12709 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.588352 32222 raft_consensus.cc:1261] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:27.589152 13454 consensus_queue.cc:1035] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.591619 13633 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.592168 32222 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.599995 29103 catalog_manager.cc:5484] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 3 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.603627 29506 raft_consensus.cc:2932] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.638550 32097 consensus_queue.cc:237] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:27.640905 29635 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:27.641028 31963 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:27.641158 32030 consensus_peers.cc:485] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768543d354744b44acdebb9a95a1579b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.641492 13454 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.642073 13454 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.645043 12709 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:27.645567 31963 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:27.650295 29635 raft_consensus.cc:2932] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:27.651252 29099 catalog_manager.cc:5484] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 3 to 4, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:27.709568 31963 consensus_queue.cc:237] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
W20240501 14:02:27.712522 31898 consensus_peers.cc:485] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c86d880c077411a99e6b9597adeeb0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.712643 29374 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:27.713212 29245 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:27.713171 13672 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.714095 12871 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.716495 12871 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:27.717337 29374 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:27.723677 29100 catalog_manager.cc:5484] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:27.726977 29245 raft_consensus.cc:2932] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:27.765518 31963 consensus_queue.cc:237] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:27.768831 32223 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:27.768879 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa4a1cdf1c214e929b1d81ae3df585d9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.769393 13691 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.769896 29374 raft_consensus.cc:1261] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:27.770419 13691 consensus_queue.cc:1035] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:27.772739 12871 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.773299 29374 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.778726 32222 raft_consensus.cc:2932] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.781852 29103 catalog_manager.cc:5484] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 3 to 4, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.814649 29245 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:27.817903 29374 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:27.818444 32222 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:27.818744 13572 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:27.819120 13479 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:27.822086 29180 consensus_peers.cc:485] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500b9c9bdfa649bd883d6ef2d6e541fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.827072 13572 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.832804 32222 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.835459 29374 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.835961 29100 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:27.852708 13713 ts_tablet_manager.cc:902] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:27.882129 13713 tablet_copy_client.cc:308] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.72:37615
I20240501 14:02:27.883634 32383 tablet_copy_service.cc:140] P 6edd6a3430d44c38a19c2804e555059f: Received BeginTabletCopySession request for tablet 3ec202bb41a44e05939f1ecdf33d8c38 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:51381)
I20240501 14:02:27.883809 32383 tablet_copy_service.cc:161] P 6edd6a3430d44c38a19c2804e555059f: Beginning new tablet copy session on tablet 3ec202bb41a44e05939f1ecdf33d8c38 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:51381: session id = 522f4ff5e9084cbf8462196533c7d7aa-3ec202bb41a44e05939f1ecdf33d8c38
I20240501 14:02:27.886768 32383 tablet_copy_source_session.cc:205] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:27.888736 13713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec202bb41a44e05939f1ecdf33d8c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.905198 13713 tablet_copy_client.cc:783] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:27.905448 13713 tablet_copy_client.cc:647] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:27.912882 13713 tablet_copy_client.cc:523] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:27.923370 13713 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:27.926599 32097 consensus_queue.cc:237] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:27.929188 32361 raft_consensus.cc:1261] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:27.929263 29245 raft_consensus.cc:1261] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:02:27.929754 32031 consensus_peers.cc:485] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 413a79f6a61a49d9a49a5a52cec4c00f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:27.929782 13454 consensus_queue.cc:1035] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:27.930289 13633 consensus_queue.cc:1035] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:27.936525 13713 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:27.936952 13713 tablet_bootstrap.cc:492] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:27.937242 13713 ts_tablet_manager.cc:1367] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240501 14:02:27.937847 13713 raft_consensus.cc:348] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:27.938046 13713 raft_consensus.cc:729] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.938252 13713 consensus_queue.cc:260] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:27.938741 13713 ts_tablet_manager.cc:1398] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:02:27.939649 32383 tablet_copy_service.cc:342] P 6edd6a3430d44c38a19c2804e555059f: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-3ec202bb41a44e05939f1ecdf33d8c38 received from {username='slave'} at 127.22.33.67:51381
I20240501 14:02:27.939785 32383 tablet_copy_service.cc:433] P 6edd6a3430d44c38a19c2804e555059f: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-3ec202bb41a44e05939f1ecdf33d8c38 on tablet 3ec202bb41a44e05939f1ecdf33d8c38 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:27.950430 13454 raft_consensus.cc:2932] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.951365 29245 raft_consensus.cc:2932] T 413a79f6a61a49d9a49a5a52cec4c00f P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.959139 32361 raft_consensus.cc:2932] T 413a79f6a61a49d9a49a5a52cec4c00f P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.961278 29103 catalog_manager.cc:5484] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index -1 to 2, VOTER 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:27.981576 13739 ts_tablet_manager.cc:902] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:28.023392 13745 ts_tablet_manager.cc:902] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:28.028481 13739 tablet_copy_client.cc:308] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:28.029438 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:47463)
I20240501 14:02:28.029592 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:47463: session id = 6edd6a3430d44c38a19c2804e555059f-54bdc21d1dc644e7ac44f5b9ad39ab32
I20240501 14:02:28.033460 32117 tablet_copy_source_session.cc:205] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.036248 13739 ts_tablet_manager.cc:1875] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:02:28.040187 13745 tablet_copy_client.cc:308] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:02:28.040942 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 578c3f4c5a7c4e2c962b64736195f0bc from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:44499)
I20240501 14:02:28.041126 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 578c3f4c5a7c4e2c962b64736195f0bc from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:44499: session id = 9903ed35bca144bcb74e9bf5102117e7-578c3f4c5a7c4e2c962b64736195f0bc
I20240501 14:02:28.044186 29266 tablet_copy_source_session.cc:205] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:02:28.046656 13745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578c3f4c5a7c4e2c962b64736195f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.048434 13739 ts_tablet_manager.cc:1888] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20240501 14:02:28.048996 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bdc21d1dc644e7ac44f5b9ad39ab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.052728 13745 tablet_copy_client.cc:783] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 5 data blocks...
I20240501 14:02:28.053774 13739 tablet_copy_client.cc:783] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.053992 13739 tablet_copy_client.cc:647] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.055938 13739 tablet_copy_client.cc:523] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.060451 29505 consensus_queue.cc:237] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.062566 13763 ts_tablet_manager.cc:902] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:28.062613 13739 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
W20240501 14:02:28.067705 29441 consensus_peers.cc:485] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:28.068084 29635 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:28.068329 13745 tablet_copy_client.cc:647] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.068764 32097 raft_consensus.cc:1261] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 522f4ff5e9084cbf8462196533c7d7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:28.068881 12416 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:28.069304 13763 tablet_copy_client.cc:308] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:28.069332 13403 consensus_queue.cc:1035] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:28.069855 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 768543d354744b44acdebb9a95a1579b from peer 6518b7a04854458bb2fb23edc8b6f7ba ({username='slave'} at 127.22.33.65:54889)
I20240501 14:02:28.069972 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 768543d354744b44acdebb9a95a1579b from peer 6518b7a04854458bb2fb23edc8b6f7ba at {username='slave'} at 127.22.33.65:54889: session id = 6518b7a04854458bb2fb23edc8b6f7ba-768543d354744b44acdebb9a95a1579b
I20240501 14:02:28.072532 32117 tablet_copy_source_session.cc:205] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.075289 13745 tablet_copy_client.cc:523] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.086057 13745 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:02:28.088961 13763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768543d354744b44acdebb9a95a1579b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.095983 13739 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.096467 13739 tablet_bootstrap.cc:492] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:02:28.096817 13739 ts_tablet_manager.cc:1367] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.005s
I20240501 14:02:28.097680 13739 raft_consensus.cc:348] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.098026 13739 raft_consensus.cc:729] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.098268 13739 consensus_queue.cc:260] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.100359 13739 ts_tablet_manager.cc:1398] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:28.100996 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-54bdc21d1dc644e7ac44f5b9ad39ab32 received from {username='slave'} at 127.22.33.72:47463
I20240501 14:02:28.101117 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-54bdc21d1dc644e7ac44f5b9ad39ab32 on tablet 54bdc21d1dc644e7ac44f5b9ad39ab32 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:28.104467 13403 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.106019 13763 tablet_copy_client.cc:783] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.106330 13763 tablet_copy_client.cc:647] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.111073 13771 ts_tablet_manager.cc:902] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:28.113273 32097 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.114483 29634 raft_consensus.cc:2932] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.120824 13763 tablet_copy_client.cc:523] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.124503 13763 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap starting.
I20240501 14:02:28.116726 29103 catalog_manager.cc:5484] T 49b72e2ac9df4426b071a7e5004e12e1 P 522f4ff5e9084cbf8462196533c7d7aa reported cstate change: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "522f4ff5e9084cbf8462196533c7d7aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.139925 13763 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.140409 13763 tablet_bootstrap.cc:492] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Bootstrap complete.
I20240501 14:02:28.140780 13763 ts_tablet_manager.cc:1367] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20240501 14:02:28.141816 13763 raft_consensus.cc:348] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:28.152422 13763 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.152874 13763 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:28.153679 13763 ts_tablet_manager.cc:1398] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20240501 14:02:28.158370 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-768543d354744b44acdebb9a95a1579b received from {username='slave'} at 127.22.33.65:54889
I20240501 14:02:28.158507 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 6518b7a04854458bb2fb23edc8b6f7ba-768543d354744b44acdebb9a95a1579b on tablet 768543d354744b44acdebb9a95a1579b with peer 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:28.162242 13745 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.162753 13745 tablet_bootstrap.cc:492] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:02:28.163093 13745 ts_tablet_manager.cc:1367] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.077s user 0.034s sys 0.004s
I20240501 14:02:28.163890 13745 raft_consensus.cc:348] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:28.164178 13745 raft_consensus.cc:729] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.164471 13745 consensus_queue.cc:260] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:28.165211 13745 ts_tablet_manager.cc:1398] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:28.167127 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-578c3f4c5a7c4e2c962b64736195f0bc received from {username='slave'} at 127.22.33.69:44499
I20240501 14:02:28.167284 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-578c3f4c5a7c4e2c962b64736195f0bc on tablet 578c3f4c5a7c4e2c962b64736195f0bc with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:28.177829 13771 tablet_copy_client.cc:308] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:02:28.178552 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet 6c86d880c077411a99e6b9597adeeb0d from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:58201)
I20240501 14:02:28.178707 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet 6c86d880c077411a99e6b9597adeeb0d from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:58201: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-6c86d880c077411a99e6b9597adeeb0d
I20240501 14:02:28.182678 31985 tablet_copy_source_session.cc:205] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.219261 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c86d880c077411a99e6b9597adeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.252017 13771 tablet_copy_client.cc:783] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.252460 13771 tablet_copy_client.cc:647] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.254096 13771 tablet_copy_client.cc:523] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.257196 13771 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:28.267220 13771 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.267581 13771 tablet_bootstrap.cc:492] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:28.267843 13771 ts_tablet_manager.cc:1367] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240501 14:02:28.268815 13771 raft_consensus.cc:348] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:28.269140 13771 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.269403 13771 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:28.270035 13771 ts_tablet_manager.cc:1398] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:28.271593 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-6c86d880c077411a99e6b9597adeeb0d received from {username='slave'} at 127.22.33.68:58201
I20240501 14:02:28.271705 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-6c86d880c077411a99e6b9597adeeb0d on tablet 6c86d880c077411a99e6b9597adeeb0d with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:28.296149 13713 ts_tablet_manager.cc:902] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:28.318224 13713 tablet_copy_client.cc:308] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.69:32783
I20240501 14:02:28.319074 31985 tablet_copy_service.cc:140] P 9903ed35bca144bcb74e9bf5102117e7: Received BeginTabletCopySession request for tablet fa4a1cdf1c214e929b1d81ae3df585d9 from peer 522f4ff5e9084cbf8462196533c7d7aa ({username='slave'} at 127.22.33.67:45945)
I20240501 14:02:28.319260 31985 tablet_copy_service.cc:161] P 9903ed35bca144bcb74e9bf5102117e7: Beginning new tablet copy session on tablet fa4a1cdf1c214e929b1d81ae3df585d9 from peer 522f4ff5e9084cbf8462196533c7d7aa at {username='slave'} at 127.22.33.67:45945: session id = 522f4ff5e9084cbf8462196533c7d7aa-fa4a1cdf1c214e929b1d81ae3df585d9
I20240501 14:02:28.322074 31985 tablet_copy_source_session.cc:205] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.323436 29370 consensus_queue.cc:237] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.325155 13739 ts_tablet_manager.cc:902] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:28.326853 29244 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:28.326894 32097 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:28.327407 12933 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:28.328372 29311 consensus_peers.cc:485] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 -> Peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Couldn't send request to peer 6edd6a3430d44c38a19c2804e555059f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71933eccec5f4fdc8b1df7111f941777. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:28.328653 12933 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:28.333112 12933 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.334115 13713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4a1cdf1c214e929b1d81ae3df585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.341640 32097 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.345573 29103 catalog_manager.cc:5484] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.350690 13713 tablet_copy_client.cc:783] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.350948 13713 tablet_copy_client.cc:647] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.352413 29244 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:28.361474 13713 tablet_copy_client.cc:523] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.364553 13713 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap starting.
I20240501 14:02:28.368144 13739 tablet_copy_client.cc:308] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:02:28.369566 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 500b9c9bdfa649bd883d6ef2d6e541fc from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:50987)
I20240501 14:02:28.369743 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 500b9c9bdfa649bd883d6ef2d6e541fc from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:50987: session id = 6edd6a3430d44c38a19c2804e555059f-500b9c9bdfa649bd883d6ef2d6e541fc
I20240501 14:02:28.371887 13713 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.372334 13713 tablet_bootstrap.cc:492] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Bootstrap complete.
I20240501 14:02:28.372560 13713 ts_tablet_manager.cc:1367] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.004s
I20240501 14:02:28.372694 29266 tablet_copy_source_session.cc:205] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.373497 13713 raft_consensus.cc:348] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:28.373731 13713 raft_consensus.cc:729] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522f4ff5e9084cbf8462196533c7d7aa, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.374588 13713 consensus_queue.cc:260] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:28.375140 13713 ts_tablet_manager.cc:1398] T fa4a1cdf1c214e929b1d81ae3df585d9 P 522f4ff5e9084cbf8462196533c7d7aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:28.377455 31985 tablet_copy_service.cc:342] P 9903ed35bca144bcb74e9bf5102117e7: Request end of tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-fa4a1cdf1c214e929b1d81ae3df585d9 received from {username='slave'} at 127.22.33.67:45945
I20240501 14:02:28.377578 31985 tablet_copy_service.cc:433] P 9903ed35bca144bcb74e9bf5102117e7: ending tablet copy session 522f4ff5e9084cbf8462196533c7d7aa-fa4a1cdf1c214e929b1d81ae3df585d9 on tablet fa4a1cdf1c214e929b1d81ae3df585d9 with peer 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:28.382694 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500b9c9bdfa649bd883d6ef2d6e541fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.388326 13739 tablet_copy_client.cc:783] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.388584 13739 tablet_copy_client.cc:647] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.390816 13739 tablet_copy_client.cc:523] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.393841 13739 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:02:28.395460 32361 raft_consensus.cc:1203] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:28.402647 32011 maintenance_manager.cc:419] P 9903ed35bca144bcb74e9bf5102117e7: Scheduling UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc): 4834 bytes on disk
I20240501 14:02:28.404443 31902 maintenance_manager.cc:643] P 9903ed35bca144bcb74e9bf5102117e7: UndoDeltaBlockGCOp(578c3f4c5a7c4e2c962b64736195f0bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240501 14:02:28.420578 29370 consensus_queue.cc:237] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:28.428391 29505 raft_consensus.cc:1203] T 3ec202bb41a44e05939f1ecdf33d8c38 P 522f4ff5e9084cbf8462196533c7d7aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:28.438575 29634 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:28.438784 31964 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:28.439163 12933 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:28.439411 12536 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:28.439710 29311 consensus_peers.cc:485] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe99c587804f47c3b58ec3ddbab8a371. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:28.453676 13739 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.454159 13739 tablet_bootstrap.cc:492] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:02:28.454440 13739 ts_tablet_manager.cc:1367] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.061s user 0.053s sys 0.004s
I20240501 14:02:28.455188 13739 raft_consensus.cc:348] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.455421 13739 raft_consensus.cc:729] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.455997 29245 raft_consensus.cc:1203] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:28.455636 13739 consensus_queue.cc:260] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.459404 13739 ts_tablet_manager.cc:1398] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:28.460158 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-500b9c9bdfa649bd883d6ef2d6e541fc received from {username='slave'} at 127.22.33.72:50987
I20240501 14:02:28.460301 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-500b9c9bdfa649bd883d6ef2d6e541fc on tablet 500b9c9bdfa649bd883d6ef2d6e541fc with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:28.462191 13713 ts_tablet_manager.cc:902] T 413a79f6a61a49d9a49a5a52cec4c00f P 522f4ff5e9084cbf8462196533c7d7aa: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
F20240501 14:02:28.462543 13713 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.queue_time_us, tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.queue_time_us) (tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.run_cpu_time_us, tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.run_cpu_time_us) (tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.run_wall_time_us, tablet-download-pool-0132590f8ed5423f983c53d9c19f95fe.run_wall_time_us) (tablet-download-pool-10e1775ccab942ff857d22e89d99764d.queue_time_us, tablet-download-pool-10e1775ccab942ff857d22e89d99764d.queue_time_us) (tablet-download-pool-10e1775ccab942ff857d22e89d99764d.run_cpu_time_us, tablet-download-pool-10e1775ccab942ff857d22e89d99764d.run_cpu_time_us) (tablet-download-pool-10e1775ccab942ff857d22e89d99764d.run_wall_time_us, tablet-download-pool-10e1775ccab942ff857d22e89d99764d.run_wall_time_us) (tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.queue_time_us, tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.queue_time_us) (tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.run_cpu_time_us, tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.run_cpu_time_us) (tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.run_wall_time_us, tablet-download-pool-3ec202bb41a44e05939f1ecdf33d8c38.run_wall_time_us) (tablet-download-pool-413a79f6a61a49d9a49a5a52cec4c00f.queue_time_us, tablet-download-pool-413a79f6a61a49d9a49a5a52cec4c00f.queue_time_us) (tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.queue_time_us, tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.queue_time_us) (tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.run_cpu_time_us, tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.run_cpu_time_us) (tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.run_wall_time_us, tablet-download-pool-54bdc21d1dc644e7ac44f5b9ad39ab32.run_wall_time_us) (tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.queue_time_us, tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.queue_time_us) (tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.run_cpu_time_us, tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.run_cpu_time_us) (tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.run_wall_time_us, tablet-download-pool-6fbb60619c1a4d8e8cd6266596a0614d.run_wall_time_us) (tablet-download-pool-8a1690e3a76644859a0711e305b8f946.queue_time_us, tablet-download-pool-8a1690e3a76644859a0711e305b8f946.queue_time_us) (tablet-download-pool-8a1690e3a76644859a0711e305b8f946.run_cpu_time_us, tablet-download-pool-8a1690e3a76644859a0711e305b8f946.run_cpu_time_us) (tablet-download-pool-8a1690e3a76644859a0711e305b8f946.run_wall_time_us, tablet-download-pool-8a1690e3a76644859a0711e305b8f946.run_wall_time_us) (tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.queue_time_us, tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.queue_time_us) (tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.run_cpu_time_us, tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.run_cpu_time_us) (tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.run_wall_time_us, tablet-download-pool-9e32147255254f2d8baff46c9b2d0e60.run_wall_time_us) (tablet-download-pool-ad7af304700545b2a28249f9f046cab6.queue_time_us, tablet-download-pool-ad7af304700545b2a28249f9f046cab6.queue_time_us) (tablet-download-pool-ad7af304700545b2a28249f9f046cab6.run_cpu_time_us, tablet-download-pool-ad7af304700545b2a28249f9f046cab6.run_cpu_time_us) (tablet-download-pool-ad7af304700545b2a28249f9f046cab6.run_wall_time_us, tablet-download-pool-ad7af304700545b2a28249f9f046cab6.run_wall_time_us) (tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.queue_time_us, tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.queue_time_us) (tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.run_cpu_time_us, tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.run_cpu_time_us) (tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.run_wall_time_us, tablet-download-pool-b12b60d5bf304c28ae2041629b8d5bdd.run_wall_time_us) (tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.queue_time_us, tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.queue_time_us) (tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.run_cpu_time_us, tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.run_cpu_time_us) (tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.run_wall_time_us, tablet-download-pool-ba3bd81448244b5bbdcd9eb8d23aee03.run_wall_time_us) (tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.queue_time_us, tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.queue_time_us) (tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.run_cpu_time_us, tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.run_cpu_time_us) (tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.run_wall_time_us, tablet-download-pool-c8020272a8b14dea8a6a82802ea4e10a.run_wall_time_us) (tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.queue_time_us, tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.queue_time_us) (tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.run_cpu_time_us, tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.run_cpu_time_us) (tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.run_wall_time_us, tablet-download-pool-cd7a7501266946dca6bc838d10524c4a.run_wall_time_us) (tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.queue_time_us, tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.queue_time_us) (tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.run_cpu_time_us, tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.run_cpu_time_us) (tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.run_wall_time_us, tablet-download-pool-fa4a1cdf1c214e929b1d81ae3df585d9.run_wall_time_us) (tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.queue_time_us, tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.queue_time_us) (tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.run_cpu_time_us, tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.run_cpu_time_us) (tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.run_wall_time_us, tablet-download-pool-fee3fe52d2ae462a823d3ac03f2ce8bc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fa68cecadcd google::LogMessage::Fail() at ??:0
@ 0x7fa68ceceb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa68ceca7cc google::LogMessage::Flush() at ??:0
@ 0x7fa68cecbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:02:28.482064 12933 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
@ 0x7fa68fc0b742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fa68fbd7f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa68fbd1cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:02:28.507105 31964 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:28.507377 29634 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
@ 0x7fa69f0248de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:02:28.513124 29103 catalog_manager.cc:5484] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 reported cstate change: config changed from index 3 to 4, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
@ 0x7fa69f03cdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fa69f167480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fa69f19fa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fa69f182981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fa69f18270e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa68fbdbb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 14:02:28.554970 31964 raft_consensus.cc:1203] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:02:28.566674 13739 ts_tablet_manager.cc:902] T 49b72e2ac9df4426b071a7e5004e12e1 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
@ 0x7fa68fbdf19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa68fbdf019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa68fbba01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa692f3f6db start_thread at ??:0
@ 0x7fa68bd7d71f clone at ??:0
I20240501 14:02:28.617177 29634 raft_consensus.cc:1203] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:28.637930 13739 tablet_copy_client.cc:308] T 49b72e2ac9df4426b071a7e5004e12e1 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.67:44187
I20240501 14:02:28.701942 29245 consensus_queue.cc:237] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:28.704628 31965 raft_consensus.cc:1261] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:28.705631 29370 raft_consensus.cc:1261] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:28.706667 13572 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:02:28.707239 29181 consensus_peers.cc:485] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3df490207f6248ce8a36687d71fedb85. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:28.707533 13572 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:28.721575 13867 ts_tablet_manager.cc:902] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:28.727687 13867 tablet_copy_client.cc:308] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:28.728541 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet 71933eccec5f4fdc8b1df7111f941777 from peer 6edd6a3430d44c38a19c2804e555059f ({username='slave'} at 127.22.33.72:49865)
I20240501 14:02:28.728680 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet 71933eccec5f4fdc8b1df7111f941777 from peer 6edd6a3430d44c38a19c2804e555059f at {username='slave'} at 127.22.33.72:49865: session id = 6edd6a3430d44c38a19c2804e555059f-71933eccec5f4fdc8b1df7111f941777
I20240501 14:02:28.731316 29397 tablet_copy_source_session.cc:205] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.734068 13572 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:28.735548 13867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71933eccec5f4fdc8b1df7111f941777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.737520 31964 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:28.744884 29102 catalog_manager.cc:5484] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:28.745999 29370 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:28.780119 13867 tablet_copy_client.cc:783] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.780383 13867 tablet_copy_client.cc:647] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.782549 13867 tablet_copy_client.cc:523] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.787679 13867 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap starting.
I20240501 14:02:28.880796 13894 ts_tablet_manager.cc:902] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:28.899333 13867 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.899784 13867 tablet_bootstrap.cc:492] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Bootstrap complete.
I20240501 14:02:28.900756 32097 consensus_queue.cc:237] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:28.906395 13867 ts_tablet_manager.cc:1367] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Time spent bootstrapping tablet: real 0.119s user 0.060s sys 0.003s
I20240501 14:02:28.907234 13867 raft_consensus.cc:348] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.907451 13867 raft_consensus.cc:729] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.908812 13894 tablet_copy_client.cc:308] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.66:39595
I20240501 14:02:28.913455 29397 tablet_copy_service.cc:140] P b87d188552494b4590afc37a85b98c35: Received BeginTabletCopySession request for tablet fe99c587804f47c3b58ec3ddbab8a371 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:57801)
I20240501 14:02:28.913669 29397 tablet_copy_service.cc:161] P b87d188552494b4590afc37a85b98c35: Beginning new tablet copy session on tablet fe99c587804f47c3b58ec3ddbab8a371 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:57801: session id = 6a446ac802a7457d9f27076781f21d79-fe99c587804f47c3b58ec3ddbab8a371
I20240501 14:02:28.916002 29397 tablet_copy_source_session.cc:205] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:28.917428 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe99c587804f47c3b58ec3ddbab8a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:28.928027 13894 tablet_copy_client.cc:783] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:28.928267 13894 tablet_copy_client.cc:647] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:28.929884 13894 tablet_copy_client.cc:523] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:28.930279 13867 consensus_queue.cc:260] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:28.932888 13867 ts_tablet_manager.cc:1398] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:02:28.933825 13894 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
I20240501 14:02:28.938936 32361 raft_consensus.cc:1203] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:02:28.945094 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6edd6a3430d44c38a19c2804e555059f-71933eccec5f4fdc8b1df7111f941777 received from {username='slave'} at 127.22.33.72:49865
I20240501 14:02:28.945314 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6edd6a3430d44c38a19c2804e555059f-71933eccec5f4fdc8b1df7111f941777 on tablet 71933eccec5f4fdc8b1df7111f941777 with peer 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:28.952579 13894 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.953059 13894 tablet_bootstrap.cc:492] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:02:28.953326 13894 ts_tablet_manager.cc:1367] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.006s
I20240501 14:02:28.954454 13894 raft_consensus.cc:348] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:28.954710 13894 raft_consensus.cc:729] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.954995 13894 consensus_queue.cc:260] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:28.955534 13894 ts_tablet_manager.cc:1398] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:28.957871 29397 tablet_copy_service.cc:342] P b87d188552494b4590afc37a85b98c35: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-fe99c587804f47c3b58ec3ddbab8a371 received from {username='slave'} at 127.22.33.71:57801
I20240501 14:02:28.957988 29397 tablet_copy_service.cc:433] P b87d188552494b4590afc37a85b98c35: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-fe99c587804f47c3b58ec3ddbab8a371 on tablet fe99c587804f47c3b58ec3ddbab8a371 with peer 6a446ac802a7457d9f27076781f21d79
W20240501 14:02:28.958029 32030 consensus_peers.cc:485] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb982ed7bcf74eb981c5199d87f9b4c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:28.961318 32361 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:28.961380 29244 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:28.961907 13633 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:28.965759 13633 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:28.970679 13454 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:28.974735 29244 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:28.994982 29103 catalog_manager.cc:5484] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:29.045039 32361 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
W20240501 14:02:29.070140 31899 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.070264 29311 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.070456 31899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.070452 29311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.070959 32162 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.071266 32162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.072640 29571 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.072659 32295 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.072904 29571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.072914 32295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.079310 29180 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.079602 29180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.080572 13920 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:29.072026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.33.67:44187 (local address 127.22.33.66:39933)
0501 14:02:29.072535 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:29.072558 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:29.072705 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:29.079904 (+ 7199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":213,"threads_started":1}
W20240501 14:02:29.080662 32031 connection.cc:535] client connection to 127.22.33.67:44187 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.080897 32031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:29.080996 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.081118 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.081202 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.081277 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.082682 13924 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:29.075053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.33.67:44187 (local address 127.22.33.68:56371)
0501 14:02:29.075478 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:29.075499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:29.075625 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:29.082049 (+ 6424us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":238,"threads_started":1}
W20240501 14:02:29.082962 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083092 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083176 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083259 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083335 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083407 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.083896 13921 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:29.072325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.33.67:44187 (local address 127.22.33.69:57141)
0501 14:02:29.075871 (+ 3546us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:29.075885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:29.075983 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:29.083265 (+ 7282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3458,"thread_start_us":3781,"threads_started":1}
W20240501 14:02:29.084111 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.088204 32297 connection.cc:441] server connection from 127.22.33.67:51381 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.22.33.67:51381 (request call id 130) could send its response
W20240501 14:02:29.090202 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.091410 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.093356 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.094453 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.097044 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.104240 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.104411 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:29.104569 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.104821 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.104962 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.105085 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.105816 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.105952 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.106042 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.114616 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.118639 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.119567 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.119879 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.120002 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.120093 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.120185 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:29.121518 13937 ts_tablet_manager.cc:902] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:29.123236 13937 tablet_copy_client.cc:308] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:02:29.124102 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet 3df490207f6248ce8a36687d71fedb85 from peer 0a697f3a1ee544daa7073814ab845d58 ({username='slave'} at 127.22.33.70:36699)
I20240501 14:02:29.124300 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet 3df490207f6248ce8a36687d71fedb85 from peer 0a697f3a1ee544daa7073814ab845d58 at {username='slave'} at 127.22.33.70:36699: session id = 0a697f3a1ee544daa7073814ab845d58-3df490207f6248ce8a36687d71fedb85
I20240501 14:02:29.126950 29266 tablet_copy_source_session.cc:205] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:29.128134 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df490207f6248ce8a36687d71fedb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:29.133914 13937 tablet_copy_client.cc:783] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:29.134166 13937 tablet_copy_client.cc:647] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:29.135716 13937 tablet_copy_client.cc:523] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:29.138731 13937 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap starting.
W20240501 14:02:29.143806 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:29.148738 32097 consensus_queue.cc:237] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.149067 13937 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:29.149492 13937 tablet_bootstrap.cc:492] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Bootstrap complete.
I20240501 14:02:29.149818 13937 ts_tablet_manager.cc:1367] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20240501 14:02:29.151051 13937 raft_consensus.cc:348] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:29.151362 13937 raft_consensus.cc:729] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Initialized, Role: FOLLOWER
I20240501 14:02:29.151633 13937 consensus_queue.cc:260] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:29.165580 13937 ts_tablet_manager.cc:1398] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 14:02:29.166718 32222 raft_consensus.cc:1261] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:29.165817 32032 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a38ce68ec4d4ba3a813bdb0ec314d4d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:29.167335 13910 consensus_queue.cc:1035] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:29.169562 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 0a697f3a1ee544daa7073814ab845d58-3df490207f6248ce8a36687d71fedb85 received from {username='slave'} at 127.22.33.70:36699
I20240501 14:02:29.169703 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 0a697f3a1ee544daa7073814ab845d58-3df490207f6248ce8a36687d71fedb85 on tablet 3df490207f6248ce8a36687d71fedb85 with peer 0a697f3a1ee544daa7073814ab845d58
I20240501 14:02:29.174731 32359 raft_consensus.cc:1203] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 14:02:29.178656 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:29.192871 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:29.217875 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(c2c63489706e4d068adde1872f451ad2): perf score=0.033361
I20240501 14:02:29.243943 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(e513634d27c5467fb49c9717c43904cd): perf score=0.033373
I20240501 14:02:29.250286 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(c2c63489706e4d068adde1872f451ad2): perf score=0.033375
I20240501 14:02:29.259826 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(c2c63489706e4d068adde1872f451ad2) complete. Timing: real 0.042s user 0.031s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1946,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2719,"threads_started":2,"wal-append.queue_time_us":1891}
I20240501 14:02:29.263229 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(192a6cc308da4a448f03717b6a439094): perf score=0.033371
I20240501 14:02:29.295758 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(e513634d27c5467fb49c9717c43904cd) complete. Timing: real 0.052s user 0.035s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1012,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":3199,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1375,"threads_started":2,"wal-append.queue_time_us":343}
I20240501 14:02:29.298187 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(e513634d27c5467fb49c9717c43904cd): 4921 bytes on disk
I20240501 14:02:29.298890 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(e513634d27c5467fb49c9717c43904cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 14:02:29.301420 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(192a6cc308da4a448f03717b6a439094): perf score=0.033386
I20240501 14:02:29.330500 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(c2c63489706e4d068adde1872f451ad2) complete. Timing: real 0.080s user 0.044s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":722,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2053,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":987,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10792,"threads_started":2,"wal-append.queue_time_us":10435}
I20240501 14:02:29.333771 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(c2c63489706e4d068adde1872f451ad2): 4834 bytes on disk
I20240501 14:02:29.334614 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(c2c63489706e4d068adde1872f451ad2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 14:02:29.344903 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(192a6cc308da4a448f03717b6a439094) complete. Timing: real 0.079s user 0.059s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1341,"drs_written":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6643,"threads_started":1,"wal-append.queue_time_us":6517}
I20240501 14:02:29.347743 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(c2c63489706e4d068adde1872f451ad2): 4834 bytes on disk
I20240501 14:02:29.348402 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(c2c63489706e4d068adde1872f451ad2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:02:29.351040 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094): 4581 bytes on disk
I20240501 14:02:29.351614 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 14:02:29.358830 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(192a6cc308da4a448f03717b6a439094) complete. Timing: real 0.057s user 0.038s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1502,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1026,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":508,"threads_started":1,"wal-append.queue_time_us":644}
I20240501 14:02:29.361158 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094): 4581 bytes on disk
I20240501 14:02:29.361752 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 14:02:29.366995 32222 raft_consensus.cc:1203] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:29.519693 32096 raft_consensus.cc:1203] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:02:29.581295 13960 ts_tablet_manager.cc:902] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:29.582592 13960 tablet_copy_client.cc:308] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:29.583181 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet eb982ed7bcf74eb981c5199d87f9b4c4 from peer ecc8bf0965164f199fb0bfa3d7b47c0e ({username='slave'} at 127.22.33.68:57709)
I20240501 14:02:29.583324 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet eb982ed7bcf74eb981c5199d87f9b4c4 from peer ecc8bf0965164f199fb0bfa3d7b47c0e at {username='slave'} at 127.22.33.68:57709: session id = ecc8bf0965164f199fb0bfa3d7b47c0e-eb982ed7bcf74eb981c5199d87f9b4c4
I20240501 14:02:29.585296 32117 tablet_copy_source_session.cc:205] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:29.586387 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb982ed7bcf74eb981c5199d87f9b4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:29.591293 13960 tablet_copy_client.cc:783] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:29.591490 13960 tablet_copy_client.cc:647] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:29.592913 13960 tablet_copy_client.cc:523] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:29.596079 13960 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap starting.
I20240501 14:02:29.603523 13960 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:29.603848 13960 tablet_bootstrap.cc:492] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Bootstrap complete.
I20240501 14:02:29.604056 13960 ts_tablet_manager.cc:1367] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240501 14:02:29.604626 13960 raft_consensus.cc:348] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:29.604825 13960 raft_consensus.cc:729] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Initialized, Role: FOLLOWER
I20240501 14:02:29.605077 13960 consensus_queue.cc:260] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:29.605566 13960 ts_tablet_manager.cc:1398] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:02:29.606317 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-eb982ed7bcf74eb981c5199d87f9b4c4 received from {username='slave'} at 127.22.33.68:57709
I20240501 14:02:29.606451 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session ecc8bf0965164f199fb0bfa3d7b47c0e-eb982ed7bcf74eb981c5199d87f9b4c4 on tablet eb982ed7bcf74eb981c5199d87f9b4c4 with peer ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:29.613147 13814 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.613335 13814 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:29.614077 13814 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:29.615312 29571 leader_election.cc:334] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.616125 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:02:29.616657 29570 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:29.616906 13814 raft_consensus.cc:2726] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:02:29.633381 13814 raft_consensus.cc:482] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.633548 13814 raft_consensus.cc:504] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:29.634192 13814 leader_election.cc:288] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:29.634687 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:02:29.635645 29571 leader_election.cc:334] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.635748 29571 leader_election.cc:302] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 0a697f3a1ee544daa7073814ab845d58, 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.636016 13814 raft_consensus.cc:2726] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.655604 13876 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.655829 13876 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.656675 13876 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.657631 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
W20240501 14:02:29.658288 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:29.659049 32162 leader_election.cc:334] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.659215 32162 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35
I20240501 14:02:29.659500 13876 raft_consensus.cc:2726] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.687011 13876 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.687214 13876 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.687990 13876 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:29.688555 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:02:29.689390 32162 leader_election.cc:334] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.689525 32162 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:29.689778 13876 raft_consensus.cc:2726] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.703119 13864 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.703337 13864 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:29.704087 13864 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.704607 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:02:29.704847 29634 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 2.
I20240501 14:02:29.705902 29179 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
W20240501 14:02:29.706120 29180 leader_election.cc:334] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.706207 13864 raft_consensus.cc:2781] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:02:29.706287 13864 raft_consensus.cc:482] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.706385 13864 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:02:29.708272 13864 raft_consensus.cc:504] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:29.708855 13864 leader_election.cc:288] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.709259 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e1775ccab942ff857d22e89d99764d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:02:29.709450 29634 raft_consensus.cc:3037] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:02:29.710186 29180 leader_election.cc:334] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.711565 29634 raft_consensus.cc:2454] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 3.
I20240501 14:02:29.712055 29179 leader_election.cc:302] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.712374 13864 raft_consensus.cc:2781] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:02:29.712536 13864 raft_consensus.cc:686] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [term 3 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:02:29.712823 13864 consensus_queue.cc:237] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:29.716959 29103 catalog_manager.cc:5484] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 2 to 3, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 3 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } }
I20240501 14:02:29.722124 13443 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.722301 13443 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:29.722967 13443 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:29.723414 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
W20240501 14:02:29.724148 32295 leader_election.cc:334] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.724247 32295 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:29.724462 13443 raft_consensus.cc:2726] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.728366 13971 ts_tablet_manager.cc:902] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:29.730121 13971 tablet_copy_client.cc:308] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.70:37745
I20240501 14:02:29.730844 32117 tablet_copy_service.cc:140] P 0a697f3a1ee544daa7073814ab845d58: Received BeginTabletCopySession request for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:40233)
I20240501 14:02:29.731005 32117 tablet_copy_service.cc:161] P 0a697f3a1ee544daa7073814ab845d58: Beginning new tablet copy session on tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:40233: session id = 9903ed35bca144bcb74e9bf5102117e7-4a38ce68ec4d4ba3a813bdb0ec314d4d
I20240501 14:02:29.733028 32117 tablet_copy_source_session.cc:205] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:29.734445 13971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:29.739970 13971 tablet_copy_client.cc:783] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:29.740217 13971 tablet_copy_client.cc:647] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:29.741034 13838 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.741191 13838 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.741844 13838 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.742537 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:02:29.742779 32222 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:02:29.742892 13971 tablet_copy_client.cc:523] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:29.743232 31899 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters:
W20240501 14:02:29.743482 31899 leader_election.cc:334] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.743747 13838 raft_consensus.cc:2781] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:29.743856 13838 raft_consensus.cc:482] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.743953 13838 raft_consensus.cc:3037] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:29.747105 13971 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:02:29.747032 13838 raft_consensus.cc:504] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.748201 13838 leader_election.cc:288] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.748239 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3d8b11a9bd4f69a855aa087e7cbb5d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:02:29.748438 32222 raft_consensus.cc:3037] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:29.751164 32222 raft_consensus.cc:2454] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
W20240501 14:02:29.751865 31899 leader_election.cc:334] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.752121 31899 leader_election.cc:302] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 9903ed35bca144bcb74e9bf5102117e7; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.752408 13965 raft_consensus.cc:2781] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:29.752557 13965 raft_consensus.cc:686] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:02:29.752795 13965 consensus_queue.cc:237] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.756629 29103 catalog_manager.cc:5484] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:29.763571 13971 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:02:29.764078 13971 tablet_bootstrap.cc:492] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:02:29.764374 13971 ts_tablet_manager.cc:1367] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20240501 14:02:29.765291 13971 raft_consensus.cc:348] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.765725 13971 raft_consensus.cc:729] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:29.765908 13971 consensus_queue.cc:260] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.766399 13971 ts_tablet_manager.cc:1398] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:29.767165 32117 tablet_copy_service.cc:342] P 0a697f3a1ee544daa7073814ab845d58: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-4a38ce68ec4d4ba3a813bdb0ec314d4d received from {username='slave'} at 127.22.33.69:40233
I20240501 14:02:29.767302 32117 tablet_copy_service.cc:433] P 0a697f3a1ee544daa7073814ab845d58: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-4a38ce68ec4d4ba3a813bdb0ec314d4d on tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d with peer 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:29.818624 13802 raft_consensus.cc:482] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.818830 13802 raft_consensus.cc:504] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:29.819713 13802 leader_election.cc:288] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:29.820168 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
W20240501 14:02:29.821206 29311 leader_election.cc:334] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.821358 29311 leader_election.cc:302] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:29.821628 13802 raft_consensus.cc:2726] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.822481 13864 raft_consensus.cc:482] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.822652 13864 raft_consensus.cc:504] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.823328 13864 leader_election.cc:288] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:29.824826 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:02:29.825053 29180 leader_election.cc:334] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.825395 29181 leader_election.cc:302] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:29.825809 13572 raft_consensus.cc:2726] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.849314 13965 raft_consensus.cc:482] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.849536 13965 raft_consensus.cc:504] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.850299 13965 leader_election.cc:288] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.850678 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:02:29.850920 29245 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:02:29.851347 31898 leader_election.cc:302] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:02:29.851596 13965 raft_consensus.cc:2781] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:29.851694 13965 raft_consensus.cc:482] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.851795 13965 raft_consensus.cc:3037] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:29.853732 31899 leader_election.cc:334] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.855002 13965 raft_consensus.cc:504] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.855583 13965 leader_election.cc:288] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.855861 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5893a54b753f4d52a900d34140f6363d" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:02:29.856069 29245 raft_consensus.cc:3037] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:29.858464 31899 leader_election.cc:334] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.859548 29245 raft_consensus.cc:2454] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:02:29.859998 31898 leader_election.cc:302] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.860478 13838 raft_consensus.cc:2781] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:29.861191 13838 raft_consensus.cc:686] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:02:29.861477 13838 consensus_queue.cc:237] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:29.869140 29102 catalog_manager.cc:5484] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:29.874222 13864 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.874398 13864 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
W20240501 14:02:29.876696 29180 leader_election.cc:334] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.876788 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:02:29.877315 13864 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.877576 29181 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:29.877907 13864 raft_consensus.cc:2726] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:29.910907 13876 raft_consensus.cc:482] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.911100 13876 raft_consensus.cc:504] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.911806 13876 leader_election.cc:288] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.912346 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:02:29.912555 29370 raft_consensus.cc:2454] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
I20240501 14:02:29.912997 32163 leader_election.cc:302] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters:
W20240501 14:02:29.913488 32162 leader_election.cc:334] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.913528 13876 raft_consensus.cc:2781] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:29.913820 13876 raft_consensus.cc:482] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.913933 13876 raft_consensus.cc:3037] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:29.916754 13876 raft_consensus.cc:504] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.918471 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8427dee24ad04247b23b24570c73c92a" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:02:29.918697 29370 raft_consensus.cc:3037] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:29.918880 32162 leader_election.cc:334] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.920826 29370 raft_consensus.cc:2454] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 2.
I20240501 14:02:29.921648 32163 leader_election.cc:302] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.921939 13282 raft_consensus.cc:2781] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:29.922981 13876 leader_election.cc:288] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:29.923457 13282 raft_consensus.cc:686] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [term 2 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:02:29.923789 13282 consensus_queue.cc:237] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.930864 29102 catalog_manager.cc:5484] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 2 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:29.959753 12915 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.959940 12915 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:29.960683 12915 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:29.961439 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:02:29.961707 32361 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:29.961961 29571 leader_election.cc:334] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.962203 29571 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.962428 12915 raft_consensus.cc:2781] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:29.962522 12915 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.962607 12915 raft_consensus.cc:3037] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:29.964515 12915 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:29.965134 12915 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:29.965893 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e5851626b54e1caf6105a70ccafc43" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:02:29.966086 32361 raft_consensus.cc:3037] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:29.966289 29571 leader_election.cc:334] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.968264 32361 raft_consensus.cc:2454] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:02:29.968634 29571 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:29.968933 12915 raft_consensus.cc:2781] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:29.969121 12915 raft_consensus.cc:686] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:02:29.969415 12915 consensus_queue.cc:237] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:29.973577 29102 catalog_manager.cc:5484] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:29.995980 13802 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.996217 13802 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:29.996886 13802 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:29.997800 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:02:29.998056 32222 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 1.
I20240501 14:02:29.998597 29311 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters:
W20240501 14:02:29.998898 29311 leader_election.cc:334] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:29.998924 13802 raft_consensus.cc:2781] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:29.999035 13802 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:29.999130 13802 raft_consensus.cc:3037] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.001156 29634 raft_consensus.cc:1203] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:30.001581 13802 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.002403 13802 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:30.003304 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffd2100f104b339bfdc2d272ba3729" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:02:30.003549 32224 raft_consensus.cc:3037] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.004618 13910 raft_consensus.cc:482] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.004817 13910 raft_consensus.cc:504] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:30.005451 13978 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.005636 13978 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:30.006317 13978 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:30.006386 32224 raft_consensus.cc:2454] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87d188552494b4590afc37a85b98c35 in term 2.
W20240501 14:02:30.006522 29311 leader_election.cc:334] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.006811 29311 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.007243 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
W20240501 14:02:30.007293 32031 leader_election.cc:334] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.007481 29634 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:02:30.008456 32030 leader_election.cc:302] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.008744 13911 raft_consensus.cc:2781] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:30.008865 13911 raft_consensus.cc:482] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.008963 13911 raft_consensus.cc:3037] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.009002 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" is_pre_election: true
I20240501 14:02:30.009224 29245 raft_consensus.cc:2454] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
W20240501 14:02:30.010025 31899 leader_election.cc:334] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.011376 13911 raft_consensus.cc:504] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:30.012058 13911 leader_election.cc:288] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Requested vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.013047 13802 raft_consensus.cc:2781] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:30.013316 13802 raft_consensus.cc:686] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [term 2 LEADER]: Becoming Leader. State: Replica: b87d188552494b4590afc37a85b98c35, State: Running, Role: LEADER
I20240501 14:02:30.013597 13802 consensus_queue.cc:237] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.013902 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c02994dfef4a10a87b1f47d3ab6e3b" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:02:30.014168 29634 raft_consensus.cc:3037] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.014607 13910 leader_election.cc:288] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.016899 13910 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.017077 13910 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:30.018015 13910 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:30.018379 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:02:30.019433 29102 catalog_manager.cc:5484] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to b87d188552494b4590afc37a85b98c35 (127.22.33.66). New cstate: current_term: 2 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:30.022704 32031 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:02:30.022899 32031 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6edd6a3430d44c38a19c2804e555059f, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:30.023209 13984 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:30.023269 32031 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:30.023542 32031 leader_election.cc:334] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.023777 12915 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.023941 12915 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:30.025707 31898 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.025996 13978 raft_consensus.cc:2781] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:30.026095 13978 raft_consensus.cc:482] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.026188 13978 raft_consensus.cc:3037] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.027110 29634 raft_consensus.cc:2454] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:02:30.028424 13978 raft_consensus.cc:504] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:30.029059 32030 leader_election.cc:302] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.029394 13978 leader_election.cc:288] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Requested vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:30.030586 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b4f0fc35a2401a9b80c3d590e6dbb8" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:02:30.030829 29245 raft_consensus.cc:3037] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.031783 13572 raft_consensus.cc:482] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.031955 13572 raft_consensus.cc:504] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:30.032568 13572 leader_election.cc:288] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.033273 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:02:30.033497 29370 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 1.
I20240501 14:02:30.033627 29245 raft_consensus.cc:2454] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:02:30.033962 29181 leader_election.cc:302] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters:
I20240501 14:02:30.034067 31898 leader_election.cc:302] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:02:30.034238 13864 raft_consensus.cc:2781] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:30.034337 13864 raft_consensus.cc:482] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.034435 13864 raft_consensus.cc:3037] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.034456 13978 raft_consensus.cc:2781] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:30.034600 13978 raft_consensus.cc:686] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:02:30.034847 13978 consensus_queue.cc:237] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
W20240501 14:02:30.036101 29180 leader_election.cc:334] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.037134 13984 raft_consensus.cc:2781] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:30.037377 13984 raft_consensus.cc:686] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:02:30.037736 13864 raft_consensus.cc:504] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:30.038262 13984 consensus_queue.cc:237] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:30.038399 13864 leader_election.cc:288] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.039291 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190940ae8b44ca2bf1ee496a9d624ba" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:02:30.039525 29370 raft_consensus.cc:3037] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:30.043687 29180 leader_election.cc:334] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.044015 29370 raft_consensus.cc:2454] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 2.
I20240501 14:02:30.044592 29181 leader_election.cc:302] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.044836 13802 raft_consensus.cc:482] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.045019 13802 raft_consensus.cc:504] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.045549 13864 raft_consensus.cc:2781] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:30.045678 13802 leader_election.cc:288] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:30.046310 13864 raft_consensus.cc:686] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:02:30.046412 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "b87d188552494b4590afc37a85b98c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:02:30.046766 13864 consensus_queue.cc:237] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
W20240501 14:02:30.047127 29311 leader_election.cc:334] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.047389 29311 leader_election.cc:302] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6a446ac802a7457d9f27076781f21d79
I20240501 14:02:30.047665 13802 raft_consensus.cc:2726] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:30.052314 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.053786 29102 catalog_manager.cc:5484] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
W20240501 14:02:30.057825 31899 leader_election.cc:334] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.063550 13282 raft_consensus.cc:482] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.063759 13282 raft_consensus.cc:504] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.064435 13282 leader_election.cc:288] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.066481 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35" is_pre_election: true
I20240501 14:02:30.066834 29370 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 1.
W20240501 14:02:30.067032 32162 leader_election.cc:334] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.067442 32163 leader_election.cc:302] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.067767 13282 raft_consensus.cc:2781] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:30.067854 13282 raft_consensus.cc:482] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:30.067936 13282 raft_consensus.cc:3037] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:30.067871 29102 catalog_manager.cc:5484] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 2 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:30.070256 13282 raft_consensus.cc:504] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.071965 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8249c87f42faa9a7464369048090" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:02:30.072314 29370 raft_consensus.cc:3037] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:30.072454 32162 leader_election.cc:334] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.073869 13282 leader_election.cc:288] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: Requested vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:30.075258 29370 raft_consensus.cc:2454] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a446ac802a7457d9f27076781f21d79 in term 2.
I20240501 14:02:30.075702 32163 leader_election.cc:302] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, b87d188552494b4590afc37a85b98c35; no voters: 522f4ff5e9084cbf8462196533c7d7aa
I20240501 14:02:30.075984 13282 raft_consensus.cc:2781] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:30.076117 13282 raft_consensus.cc:686] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [term 2 LEADER]: Becoming Leader. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Running, Role: LEADER
I20240501 14:02:30.076406 13282 consensus_queue.cc:237] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.085184 29102 catalog_manager.cc:5484] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
I20240501 14:02:30.088496 12915 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:30.089977 29571 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:02:30.090188 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:02:30.090417 32096 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:30.090852 29571 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.090970 29571 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:30.091349 12915 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:30.094993 29102 catalog_manager.cc:5484] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: term changed from 1 to 2, leader changed from 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67) to 6a446ac802a7457d9f27076781f21d79 (127.22.33.71). New cstate: current_term: 2 leader_uuid: "6a446ac802a7457d9f27076781f21d79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:30.183704 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.199337 29634 raft_consensus.cc:1261] T 10e1775ccab942ff857d22e89d99764d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 3 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:02:30.199875 13864 consensus_queue.cc:1035] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:30.201339 32222 raft_consensus.cc:1261] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:30.204303 13978 consensus_queue.cc:1035] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:30.215234 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.330030 31963 raft_consensus.cc:1203] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:30.330888 13911 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.331388 31963 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.334124 13911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:44930 (request call id 0) took 1186 ms. Trace:
I20240501 14:02:30.334312 13911 rpcz_store.cc:276] 0501 14:02:29.147974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:02:29.148026 (+ 52us) service_pool.cc:225] Handling call
0501 14:02:29.149702 (+ 1676us) log.cc:842] Serialized 482 byte log entry
0501 14:02:30.334072 (+1184370us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":105,"raft.run_cpu_time_us":595,"raft.run_wall_time_us":751,"spinlock_wait_cycles":591744,"thread_start_us":14843,"threads_started":1,"wal-append.queue_time_us":14935,"wal-append.run_cpu_time_us":599,"wal-append.run_wall_time_us":1000646}
I20240501 14:02:30.339007 32222 raft_consensus.cc:2932] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.340782 29102 catalog_manager.cc:5484] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 3 to 4, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.385859 29245 raft_consensus.cc:1261] T 5893a54b753f4d52a900d34140f6363d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:30.386859 13965 consensus_queue.cc:1035] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:30.393710 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.394532 29634 consensus_queue.cc:237] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:30.397465 29370 raft_consensus.cc:1261] T 8427dee24ad04247b23b24570c73c92a P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:30.398054 13282 consensus_queue.cc:1035] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:30.408537 32222 raft_consensus.cc:1261] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:30.409346 13802 consensus_queue.cc:1035] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:30.411168 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.415935 31963 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:30.416568 29370 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:30.416704 13814 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:30.417099 29571 consensus_peers.cc:485] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Couldn't send request to peer 6a446ac802a7457d9f27076781f21d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 192a6cc308da4a448f03717b6a439094. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.417716 13814 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:30.430884 13814 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:30.432590 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.439824 31963 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:30.454543 29102 catalog_manager.cc:5484] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: config changed from index 5 to 6, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:30.458853 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:30.472870 32359 raft_consensus.cc:1261] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240501 14:02:30.474438 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.477140 12915 consensus_queue.cc:1035] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:30.506199 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:30.507300 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.534538 29370 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a446ac802a7457d9f27076781f21d79 (127.22.33.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:30.541970 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.553246 29370 raft_consensus.cc:1261] T e190940ae8b44ca2bf1ee496a9d624ba P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:30.553920 13864 consensus_queue.cc:1035] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:30.560496 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.578554 29245 raft_consensus.cc:1261] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer 9903ed35bca144bcb74e9bf5102117e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:02:30.579600 13965 consensus_queue.cc:1035] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:02:30.617355 29245 consensus_queue.cc:237] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:30.619729 32097 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:30.619861 29370 raft_consensus.cc:1261] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:02:30.620127 29181 consensus_peers.cc:485] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6973301bdd94c2b8f98e718715ff752. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.620476 13990 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:30.620805 13864 consensus_queue.cc:1035] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:30.625523 29374 raft_consensus.cc:1261] T 5a8d8249c87f42faa9a7464369048090 P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 6a446ac802a7457d9f27076781f21d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:30.626703 13980 consensus_queue.cc:1035] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:30.627996 13990 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.630522 29374 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
W20240501 14:02:30.632624 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:30.637326 32096 raft_consensus.cc:2932] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.642628 29102 catalog_manager.cc:5484] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 5 to 6, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) added. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:30.650179 29634 raft_consensus.cc:1261] T 21c02994dfef4a10a87b1f47d3ab6e3b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:30.650758 13910 consensus_queue.cc:1035] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:02:30.893121 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:31.036710 14107 ts_tablet_manager.cc:902] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Initiating tablet copy from peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:31.046370 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.094321 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.101027 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.104507 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.115566 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.115769 14107 tablet_copy_client.cc:308] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.68:34947
I20240501 14:02:31.116470 29657 tablet_copy_service.cc:140] P ecc8bf0965164f199fb0bfa3d7b47c0e: Received BeginTabletCopySession request for tablet 192a6cc308da4a448f03717b6a439094 from peer 6a446ac802a7457d9f27076781f21d79 ({username='slave'} at 127.22.33.71:59115)
I20240501 14:02:31.116597 29657 tablet_copy_service.cc:161] P ecc8bf0965164f199fb0bfa3d7b47c0e: Beginning new tablet copy session on tablet 192a6cc308da4a448f03717b6a439094 from peer 6a446ac802a7457d9f27076781f21d79 at {username='slave'} at 127.22.33.71:59115: session id = 6a446ac802a7457d9f27076781f21d79-192a6cc308da4a448f03717b6a439094
I20240501 14:02:31.120791 29657 tablet_copy_source_session.cc:205] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:02:31.127009 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192a6cc308da4a448f03717b6a439094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:31.127159 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:31.152678 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.156287 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:31.163210 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:31.181905 14107 tablet_copy_client.cc:783] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 5 data blocks...
I20240501 14:02:31.192504 14123 ts_tablet_manager.cc:902] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Initiating tablet copy from peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
W20240501 14:02:31.197674 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.206569 14107 tablet_copy_client.cc:647] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Starting download of 1 WAL segments...
W20240501 14:02:31.223699 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.237111 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.240402 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.243162 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:31.243328 14107 tablet_copy_client.cc:523] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:31.251765 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:31.258517 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.270071 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.284008 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:31.310297 14107 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Bootstrap starting.
W20240501 14:02:31.331180 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.338066 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:02:31.341048 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:31.346552 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.358196 14123 tablet_copy_client.cc:308] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.33.65:43201
I20240501 14:02:31.358968 29266 tablet_copy_service.cc:140] P 6518b7a04854458bb2fb23edc8b6f7ba: Received BeginTabletCopySession request for tablet c6973301bdd94c2b8f98e718715ff752 from peer 9903ed35bca144bcb74e9bf5102117e7 ({username='slave'} at 127.22.33.69:44499)
I20240501 14:02:31.359295 29266 tablet_copy_service.cc:161] P 6518b7a04854458bb2fb23edc8b6f7ba: Beginning new tablet copy session on tablet c6973301bdd94c2b8f98e718715ff752 from peer 9903ed35bca144bcb74e9bf5102117e7 at {username='slave'} at 127.22.33.69:44499: session id = 9903ed35bca144bcb74e9bf5102117e7-c6973301bdd94c2b8f98e718715ff752
I20240501 14:02:31.362668 29266 tablet_copy_source_session.cc:205] T c6973301bdd94c2b8f98e718715ff752 P 6518b7a04854458bb2fb23edc8b6f7ba: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:02:31.364163 14123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6973301bdd94c2b8f98e718715ff752. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:02:31.364209 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.371922 14123 tablet_copy_client.cc:783] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 5 data blocks...
I20240501 14:02:31.390738 14123 tablet_copy_client.cc:647] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Starting download of 1 WAL segments...
W20240501 14:02:31.392844 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.395009 14123 tablet_copy_client.cc:523] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:31.398746 14123 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap starting.
I20240501 14:02:31.417590 14107 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:31.418159 14107 tablet_bootstrap.cc:492] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Bootstrap complete.
I20240501 14:02:31.418502 14107 ts_tablet_manager.cc:1367] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Time spent bootstrapping tablet: real 0.108s user 0.024s sys 0.001s
I20240501 14:02:31.419404 14107 raft_consensus.cc:348] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:31.419670 14107 raft_consensus.cc:729] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a446ac802a7457d9f27076781f21d79, State: Initialized, Role: FOLLOWER
I20240501 14:02:31.419994 14107 consensus_queue.cc:260] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:31.420589 14107 ts_tablet_manager.cc:1398] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:31.421269 14123 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:31.421744 14123 tablet_bootstrap.cc:492] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Bootstrap complete.
I20240501 14:02:31.422101 14123 ts_tablet_manager.cc:1367] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20240501 14:02:31.423686 14123 raft_consensus.cc:348] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:31.424108 14123 raft_consensus.cc:729] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:31.425627 29657 tablet_copy_service.cc:342] P ecc8bf0965164f199fb0bfa3d7b47c0e: Request end of tablet copy session 6a446ac802a7457d9f27076781f21d79-192a6cc308da4a448f03717b6a439094 received from {username='slave'} at 127.22.33.71:59115
I20240501 14:02:31.425774 29657 tablet_copy_service.cc:433] P ecc8bf0965164f199fb0bfa3d7b47c0e: ending tablet copy session 6a446ac802a7457d9f27076781f21d79-192a6cc308da4a448f03717b6a439094 on tablet 192a6cc308da4a448f03717b6a439094 with peer 6a446ac802a7457d9f27076781f21d79
I20240501 14:02:31.424546 14123 consensus_queue.cc:260] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:31.432721 14123 ts_tablet_manager.cc:1398] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:02:31.437366 29266 tablet_copy_service.cc:342] P 6518b7a04854458bb2fb23edc8b6f7ba: Request end of tablet copy session 9903ed35bca144bcb74e9bf5102117e7-c6973301bdd94c2b8f98e718715ff752 received from {username='slave'} at 127.22.33.69:44499
I20240501 14:02:31.437505 29266 tablet_copy_service.cc:433] P 6518b7a04854458bb2fb23edc8b6f7ba: ending tablet copy session 9903ed35bca144bcb74e9bf5102117e7-c6973301bdd94c2b8f98e718715ff752 on tablet c6973301bdd94c2b8f98e718715ff752 with peer 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:02:31.440160 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.442781 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.443107 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.470491 32011 maintenance_manager.cc:419] P 9903ed35bca144bcb74e9bf5102117e7: Scheduling UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752): 4834 bytes on disk
I20240501 14:02:31.471335 31902 maintenance_manager.cc:643] P 9903ed35bca144bcb74e9bf5102117e7: UndoDeltaBlockGCOp(c6973301bdd94c2b8f98e718715ff752) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20240501 14:02:31.477211 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.484414 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.494479 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.511514 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.520658 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.522361 32163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.69:32783, user_credentials={real_user=slave}} blocked reactor thread for 45485.2us
I20240501 14:02:31.526719 32275 maintenance_manager.cc:419] P 6a446ac802a7457d9f27076781f21d79: Scheduling UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094): 4581 bytes on disk
I20240501 14:02:31.528138 32165 maintenance_manager.cc:643] P 6a446ac802a7457d9f27076781f21d79: UndoDeltaBlockGCOp(192a6cc308da4a448f03717b6a439094) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240501 14:02:31.532544 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.575449 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.585798 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.585950 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.586127 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.588333 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.589826 32222 raft_consensus.cc:1203] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:02:31.648312 31965 raft_consensus.cc:1203] T c6973301bdd94c2b8f98e718715ff752 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 14:02:31.672959 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:31.690801 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.719414 14025 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:31.719666 14025 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:31.720598 14025 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:31.721102 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:02:31.721350 32096 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:31.722965 29571 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
W20240501 14:02:31.723148 29571 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:31.723265 29571 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:31.724282 14025 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:31.727829 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:31.810392 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:31.813869 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:31.888620 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:31.937768 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:31.967985 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:31.968251 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:32.006119 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:32.112697 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:02:32.184899 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:32.443408 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:32.449405 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:32.476356 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(8b925d954c9143279ba69ae5c90d3266): perf score=0.033356
W20240501 14:02:32.508917 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:32.529323 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:32.561738 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:32.574424 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:32.606554 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:32.618461 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:32.625748 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:02:32.655373 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(71933eccec5f4fdc8b1df7111f941777): perf score=0.033403
I20240501 14:02:32.666440 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(a82cbc0696fe4a8a815eaf0538b5aad8): perf score=0.033400
W20240501 14:02:32.688907 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:32.772224 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(a82cbc0696fe4a8a815eaf0538b5aad8) complete. Timing: real 0.105s user 0.035s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":394,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":3999,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5833,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":582,"threads_started":2,"wal-append.queue_time_us":455}
I20240501 14:02:32.775101 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(a82cbc0696fe4a8a815eaf0538b5aad8): 4581 bytes on disk
I20240501 14:02:32.780009 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(a82cbc0696fe4a8a815eaf0538b5aad8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4}
W20240501 14:02:32.783411 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:32.799917 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:32.801136 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(71933eccec5f4fdc8b1df7111f941777) complete. Timing: real 0.144s user 0.037s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":647,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1462,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9974,"threads_started":2,"wal-append.queue_time_us":6503}
I20240501 14:02:32.804514 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(71933eccec5f4fdc8b1df7111f941777): 4921 bytes on disk
I20240501 14:02:32.806659 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(71933eccec5f4fdc8b1df7111f941777) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:02:32.810500 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(a82cbc0696fe4a8a815eaf0538b5aad8): perf score=0.033435
I20240501 14:02:32.800462 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(8b925d954c9143279ba69ae5c90d3266) complete. Timing: real 0.324s user 0.044s sys 0.042s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":24611,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":3514,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":42269,"threads_started":2,"wal-append.queue_time_us":17605}
I20240501 14:02:32.815064 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(8b925d954c9143279ba69ae5c90d3266): 4834 bytes on disk
I20240501 14:02:32.815761 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(8b925d954c9143279ba69ae5c90d3266) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 14:02:32.821321 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(71933eccec5f4fdc8b1df7111f941777): perf score=0.033439
I20240501 14:02:32.876907 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(a82cbc0696fe4a8a815eaf0538b5aad8) complete. Timing: real 0.066s user 0.038s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1856,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1015,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":4992,"thread_start_us":4373,"threads_started":1,"wal-append.queue_time_us":4291}
I20240501 14:02:32.880240 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(a82cbc0696fe4a8a815eaf0538b5aad8): 4581 bytes on disk
I20240501 14:02:32.880918 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(a82cbc0696fe4a8a815eaf0538b5aad8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 14:02:32.897308 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(71933eccec5f4fdc8b1df7111f941777) complete. Timing: real 0.076s user 0.044s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1762,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":880,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":244,"threads_started":1,"wal-append.queue_time_us":373}
I20240501 14:02:32.900272 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(71933eccec5f4fdc8b1df7111f941777): 4921 bytes on disk
I20240501 14:02:32.900954 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(71933eccec5f4fdc8b1df7111f941777) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240501 14:02:32.913817 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:32.939697 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:32.944880 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:32.957240 29374 consensus_queue.cc:237] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:32.959311 29634 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:32.959355 32224 raft_consensus.cc:1261] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:32.959909 13802 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:32.960229 14098 consensus_queue.cc:1035] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:32.962775 13802 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:32.963917 29634 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:32.964043 32224 raft_consensus.cc:2932] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:32.972474 29096 catalog_manager.cc:5484] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 4 to 5, VOTER 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:32.982623 31944 tablet_service.cc:1514] Processing DeleteTablet for tablet fe99c587804f47c3b58ec3ddbab8a371 with delete_type TABLET_DATA_TOMBSTONED (TS 9903ed35bca144bcb74e9bf5102117e7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37626
I20240501 14:02:32.983280 14288 tablet_replica.cc:330] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: stopping tablet replica
I20240501 14:02:32.983616 14288 raft_consensus.cc:2229] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:32.983976 14288 raft_consensus.cc:2258] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:32.984696 14288 ts_tablet_manager.cc:1875] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:32.989864 14288 ts_tablet_manager.cc:1888] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:32.990360 14288 log.cc:1196] T fe99c587804f47c3b58ec3ddbab8a371 P 9903ed35bca144bcb74e9bf5102117e7: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe99c587804f47c3b58ec3ddbab8a371
W20240501 14:02:32.990923 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:32.991461 29091 catalog_manager.cc:4830] TS 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): tablet fe99c587804f47c3b58ec3ddbab8a371 (table rebalancer_extra_table_36 [id=c437ad9a367549aebe99d225bb56580f]) successfully deleted
W20240501 14:02:33.019491 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:33.074306 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:33.074671 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:33.156435 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:33.186396 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.273882 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:33.288601 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:33.305449 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:33.320330 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:33.324040 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:02:33.342999 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:33.394917 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:33.400319 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.477808 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.517764 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:33.535558 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.544262 14252 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:33.544445 14252 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:33.545289 14252 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:33.548044 32031 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
W20240501 14:02:33.549750 32031 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:33.549878 32031 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:33.550493 14252 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:33.552376 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:02:33.552587 29635 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
W20240501 14:02:33.565848 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.589005 29370 consensus_queue.cc:237] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:33.590575 32096 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:33.590653 32361 raft_consensus.cc:1261] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer b87d188552494b4590afc37a85b98c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:33.591094 13802 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:33.591428 14098 consensus_queue.cc:1035] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:33.594514 13802 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P b87d188552494b4590afc37a85b98c35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:33.595067 32096 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:33.597545 32361 raft_consensus.cc:2932] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:33.602227 29100 catalog_manager.cc:5484] T 71933eccec5f4fdc8b1df7111f941777 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: config changed from index 6 to 7, VOTER 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) evicted. New cstate: current_term: 1 leader_uuid: "b87d188552494b4590afc37a85b98c35" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:33.615263 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet 71933eccec5f4fdc8b1df7111f941777 with delete_type TABLET_DATA_TOMBSTONED (TS 6518b7a04854458bb2fb23edc8b6f7ba not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48910
I20240501 14:02:33.616604 14374 tablet_replica.cc:330] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: stopping tablet replica
I20240501 14:02:33.616849 14374 raft_consensus.cc:2229] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:33.617168 14374 raft_consensus.cc:2258] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:33.618697 14374 ts_tablet_manager.cc:1875] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:02:33.624655 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.630376 14374 ts_tablet_manager.cc:1888] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:02:33.630534 14374 log.cc:1196] T 71933eccec5f4fdc8b1df7111f941777 P 6518b7a04854458bb2fb23edc8b6f7ba: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/wals/71933eccec5f4fdc8b1df7111f941777
I20240501 14:02:33.632823 29089 catalog_manager.cc:4830] TS 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): tablet 71933eccec5f4fdc8b1df7111f941777 (table rebalance_test_table_15 [id=a9d691283009410b92557f2186334223]) successfully deleted
W20240501 14:02:33.662017 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.669706 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.679498 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.696666 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.763026 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.791131 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.841902 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.870414 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.885578 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:33.901540 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:33.924360 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.938544 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:33.956238 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:02:33.994411 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:33.994659 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:34.008023 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:34.013928 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:34.015313 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.024812 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:34.061203 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:34.063867 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.067565 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.068822 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.068921 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.080407 32295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 14:02:34.082036 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.085218 31899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 14:02:34.085678 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:34.096685 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:02:34.101473 32031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:02:34.101505 29571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:02:34.102573 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:34.149485 29180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:02:34.168463 29311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240501 14:02:34.170854 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.206661 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.213141 32162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240501 14:02:34.214198 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.228745 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:34.257180 29634 consensus_queue.cc:237] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:34.259218 31965 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:34.259281 32222 raft_consensus.cc:1261] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:34.259704 14261 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:34.260135 14025 consensus_queue.cc:1035] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:34.263728 14025 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:34.264094 32224 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:34.270053 31965 raft_consensus.cc:2932] T 192a6cc308da4a448f03717b6a439094 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
I20240501 14:02:34.270339 29099 catalog_manager.cc:5484] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } }
W20240501 14:02:34.271968 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.276860 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:34.277987 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:34.287277 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 192a6cc308da4a448f03717b6a439094 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:34.290161 14468 tablet_replica.cc:330] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:34.290421 14468 raft_consensus.cc:2229] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:34.290835 14468 raft_consensus.cc:2258] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:34.291947 14468 ts_tablet_manager.cc:1875] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:34.309279 14468 ts_tablet_manager.cc:1888] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:02:34.309453 14468 log.cc:1196] T 192a6cc308da4a448f03717b6a439094 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/192a6cc308da4a448f03717b6a439094
I20240501 14:02:34.310369 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 192a6cc308da4a448f03717b6a439094 (table rebalance_test_table_13 [id=35020b4bdfd14884929b6c060344c01f]) successfully deleted
W20240501 14:02:34.581912 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:34.608318 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:34.679315 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:34.699564 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:34.700424 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:34.731693 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:34.738682 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:02:34.762444 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:34.816998 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(60527f81f34b45c199bf52bf0df53ee2): perf score=0.033353
I20240501 14:02:34.876376 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(60527f81f34b45c199bf52bf0df53ee2) complete. Timing: real 0.059s user 0.039s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4044,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":768,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3718,"threads_started":2,"wal-append.queue_time_us":717}
I20240501 14:02:34.878741 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(ba20c01d741a414196972ced1601baf5): perf score=0.033367
I20240501 14:02:34.907552 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(5a8d8249c87f42faa9a7464369048090): perf score=0.033375
I20240501 14:02:34.931531 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(60527f81f34b45c199bf52bf0df53ee2): perf score=0.033372
I20240501 14:02:34.978799 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(ba20c01d741a414196972ced1601baf5) complete. Timing: real 0.100s user 0.055s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3727,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6469,"threads_started":1,"wal-append.queue_time_us":6335}
I20240501 14:02:34.982625 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(ba20c01d741a414196972ced1601baf5): 4922 bytes on disk
I20240501 14:02:34.983686 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(ba20c01d741a414196972ced1601baf5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240501 14:02:34.987505 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(60527f81f34b45c199bf52bf0df53ee2): 4581 bytes on disk
I20240501 14:02:34.988406 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(60527f81f34b45c199bf52bf0df53ee2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
W20240501 14:02:34.998119 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:35.042531 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(60527f81f34b45c199bf52bf0df53ee2) complete. Timing: real 0.111s user 0.062s sys 0.031s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7691,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":6483,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":56837,"threads_started":2,"wal-append.queue_time_us":49144}
I20240501 14:02:35.045133 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(60527f81f34b45c199bf52bf0df53ee2): 4581 bytes on disk
I20240501 14:02:35.047936 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(5a8d8249c87f42faa9a7464369048090) complete. Timing: real 0.140s user 0.036s sys 0.039s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47962,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":10842,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":971,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77483,"threads_started":2,"wal-append.queue_time_us":29480}
I20240501 14:02:35.048758 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(60527f81f34b45c199bf52bf0df53ee2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:02:35.050536 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(5a8d8249c87f42faa9a7464369048090): 4834 bytes on disk
I20240501 14:02:35.053795 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(5a8d8249c87f42faa9a7464369048090) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":249,"lbm_reads_lt_1ms":4}
W20240501 14:02:35.079793 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:35.098589 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:35.144928 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:35.220081 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:35.302562 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:02:35.328791 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:35.344595 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:35.361757 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:35.363319 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:35.365654 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:35.388492 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:02:35.399066 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:35.422842 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:35.427203 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:35.488495 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:35.530440 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:35.573849 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:35.660632 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:35.694784 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:02:35.737233 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:35.812909 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:35.826112 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:02:35.857893 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:35.861223 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:35.885695 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:35.924448 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:02:35.936096 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:35.937332 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:02:35.966504 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:35.977471 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:36.078855 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.100632 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.101578 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.111011 29245 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
W20240501 14:02:36.111155 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.113338 32359 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:36.113380 32222 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:36.113951 13990 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:36.114244 14099 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:36.117544 14099 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:36.118106 32359 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:36.118281 32222 raft_consensus.cc:2932] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:36.130358 29103 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 reported cstate change: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
W20240501 14:02:36.138993 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:36.180450 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 500b9c9bdfa649bd883d6ef2d6e541fc with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:36.190358 14686 tablet_replica.cc:330] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:36.190677 14686 raft_consensus.cc:2229] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:36.190948 14686 raft_consensus.cc:2258] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:02:36.191994 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.193166 14686 ts_tablet_manager.cc:1875] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:36.210678 14686 ts_tablet_manager.cc:1888] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:02:36.210932 14686 log.cc:1196] T 500b9c9bdfa649bd883d6ef2d6e541fc P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/500b9c9bdfa649bd883d6ef2d6e541fc
I20240501 14:02:36.212052 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 500b9c9bdfa649bd883d6ef2d6e541fc (table rebalance_test_table_19 [id=ce67b4011598474bad7e6571bf51640b]) successfully deleted
W20240501 14:02:36.266590 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.305025 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.311256 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.413635 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.440876 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.441071 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.454304 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.531630 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:02:36.544358 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.544816 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:36.579253 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:36.586086 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.600798 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:36.602978 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.603843 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:02:36.610576 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.621999 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.625685 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.628923 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:36.638773 32295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.71:38743, user_credentials={real_user=slave}} blocked reactor thread for 75871.5us
W20240501 14:02:36.641615 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.642483 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.664786 14097 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:36.665019 14097 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:36.665848 14097 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:36.667106 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:36.669867 29634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" is_pre_election: true
I20240501 14:02:36.670111 29634 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
W20240501 14:02:36.671052 32031 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:02:36.671626 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:02:36.672876 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:36.678243 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.694074 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.694095 32031 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:36.694231 32031 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:36.694715 14097 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:36.740909 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.746654 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.819578 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.959672 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:36.998747 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:37.017395 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:37.034541 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:37.104452 32076 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:02:37.118059 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:37.132275 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:37.132853 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:37.134582 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:37.142832 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:37.226086 29617 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:02:37.300225 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:37.355604 32096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "768543d354744b44acdebb9a95a1579b"
dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45086
I20240501 14:02:37.355906 32096 raft_consensus.cc:595] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:37.374374 14097 raft_consensus.cc:979] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58: : Instructing follower ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:02:37.374576 14679 raft_consensus.cc:1067] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Signalling peer ecc8bf0965164f199fb0bfa3d7b47c0e to start an election
I20240501 14:02:37.375267 29634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "768543d354744b44acdebb9a95a1579b"
dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
from {username='slave'} at 127.22.33.70:48851
I20240501 14:02:37.375432 29634 raft_consensus.cc:482] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:37.375519 29634 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:37.378108 29634 raft_consensus.cc:504] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:37.378909 29634 leader_election.cc:288] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:37.379171 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:02:37.379366 31965 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:37.379595 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
I20240501 14:02:37.379753 29245 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:37.381306 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768543d354744b44acdebb9a95a1579b" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a697f3a1ee544daa7073814ab845d58"
I20240501 14:02:37.381506 32096 raft_consensus.cc:3032] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:37.381592 32096 raft_consensus.cc:729] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:02:37.381721 29245 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:02:37.381897 32096 consensus_queue.cc:260] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:37.382365 32096 raft_consensus.cc:3037] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:37.384958 32096 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:02:37.385380 29572 leader_election.cc:302] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
I20240501 14:02:37.386377 14025 raft_consensus.cc:2781] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:37.386513 14025 raft_consensus.cc:686] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 LEADER]: Becoming Leader. State: Replica: ecc8bf0965164f199fb0bfa3d7b47c0e, State: Running, Role: LEADER
I20240501 14:02:37.386706 31965 raft_consensus.cc:2454] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 2.
I20240501 14:02:37.386794 14025 consensus_queue.cc:237] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:37.391747 29105 catalog_manager.cc:5484] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e reported cstate change: term changed from 1 to 2, leader changed from 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70) to ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68). New cstate: current_term: 2 leader_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } }
W20240501 14:02:37.486583 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:37.538602 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:37.580838 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:37.626451 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:37.658186 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:37.666846 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:37.692092 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:37.729918 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:37.751559 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:37.769487 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:37.781457 29245 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:37.782163 14452 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.811214 32096 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:37.811753 14025 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:37.816190 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:37.820181 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:37.830878 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:37.845794 31965 raft_consensus.cc:1261] T 768543d354744b44acdebb9a95a1579b P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer ecc8bf0965164f199fb0bfa3d7b47c0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:37.846853 14025 consensus_queue.cc:1035] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:37.988020 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:38.022626 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:38.027979 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:38.075132 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:38.274434 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling FlushMRSOp(0e33e4f9b1204797aeff0bca36c93911): perf score=0.033368
W20240501 14:02:38.279747 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.291599 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:38.349666 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:38.352296 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling FlushMRSOp(0e33e4f9b1204797aeff0bca36c93911): perf score=0.033381
I20240501 14:02:38.355039 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:38.360888 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(500b9c9bdfa649bd883d6ef2d6e541fc): perf score=0.033397
W20240501 14:02:38.363998 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.375277 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:38.397047 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.410307 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:02:38.432427 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.462199 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.465229 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:38.477815 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:38.482071 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:02:38.485805 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:38.517872 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:38.528743 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.565788 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: FlushMRSOp(0e33e4f9b1204797aeff0bca36c93911) complete. Timing: real 0.291s user 0.049s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8990,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":37141,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1003,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97343,"threads_started":2,"wal-append.queue_time_us":50098}
I20240501 14:02:38.568608 29683 maintenance_manager.cc:419] P ecc8bf0965164f199fb0bfa3d7b47c0e: Scheduling UndoDeltaBlockGCOp(0e33e4f9b1204797aeff0bca36c93911): 4581 bytes on disk
I20240501 14:02:38.571887 29574 maintenance_manager.cc:643] P ecc8bf0965164f199fb0bfa3d7b47c0e: UndoDeltaBlockGCOp(0e33e4f9b1204797aeff0bca36c93911) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 14:02:38.630636 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(500b9c9bdfa649bd883d6ef2d6e541fc) complete. Timing: real 0.269s user 0.032s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36011,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":3604,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":959,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":60061,"threads_started":2,"wal-append.queue_time_us":319}
I20240501 14:02:38.631639 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: FlushMRSOp(0e33e4f9b1204797aeff0bca36c93911) complete. Timing: real 0.278s user 0.044s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37246,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":5186,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":84780,"threads_started":2,"wal-append.queue_time_us":43650}
I20240501 14:02:38.633334 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling FlushMRSOp(5893a54b753f4d52a900d34140f6363d): perf score=0.033462
I20240501 14:02:38.635808 29423 maintenance_manager.cc:419] P b87d188552494b4590afc37a85b98c35: Scheduling UndoDeltaBlockGCOp(0e33e4f9b1204797aeff0bca36c93911): 4581 bytes on disk
I20240501 14:02:38.636396 29314 maintenance_manager.cc:643] P b87d188552494b4590afc37a85b98c35: UndoDeltaBlockGCOp(0e33e4f9b1204797aeff0bca36c93911) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 14:02:38.704239 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: FlushMRSOp(5893a54b753f4d52a900d34140f6363d) complete. Timing: real 0.063s user 0.033s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1649,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1029,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":808,"threads_started":1,"wal-append.queue_time_us":892}
I20240501 14:02:38.707003 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(500b9c9bdfa649bd883d6ef2d6e541fc): 4921 bytes on disk
I20240501 14:02:38.707686 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(500b9c9bdfa649bd883d6ef2d6e541fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240501 14:02:38.709699 29292 maintenance_manager.cc:419] P 6518b7a04854458bb2fb23edc8b6f7ba: Scheduling UndoDeltaBlockGCOp(5893a54b753f4d52a900d34140f6363d): 4834 bytes on disk
I20240501 14:02:38.710435 29183 maintenance_manager.cc:643] P 6518b7a04854458bb2fb23edc8b6f7ba: UndoDeltaBlockGCOp(5893a54b753f4d52a900d34140f6363d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 14:02:38.718353 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.719936 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.722783 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.728885 32095 consensus_queue.cc:237] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:38.731153 29635 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.731130 29246 raft_consensus.cc:1261] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Refusing update from remote peer 0a697f3a1ee544daa7073814ab845d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.731789 14679 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.732105 14861 consensus_queue.cc:1035] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.735270 14097 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:38.735661 29635 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:38.736076 29246 raft_consensus.cc:2932] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
W20240501 14:02:38.742976 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:38.745009 29105 catalog_manager.cc:5484] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 4 to 5, VOTER 6edd6a3430d44c38a19c2804e555059f (127.22.33.72) evicted. New cstate: current_term: 1 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:38.754346 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet eb982ed7bcf74eb981c5199d87f9b4c4 with delete_type TABLET_DATA_TOMBSTONED (TS 6edd6a3430d44c38a19c2804e555059f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41308
I20240501 14:02:38.761327 14965 tablet_replica.cc:330] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: stopping tablet replica
I20240501 14:02:38.761495 14965 raft_consensus.cc:2229] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:38.761695 14965 raft_consensus.cc:2258] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:38.762928 14965 ts_tablet_manager.cc:1875] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:38.769697 14965 ts_tablet_manager.cc:1888] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:38.769815 14965 log.cc:1196] T eb982ed7bcf74eb981c5199d87f9b4c4 P 6edd6a3430d44c38a19c2804e555059f: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/wals/eb982ed7bcf74eb981c5199d87f9b4c4
I20240501 14:02:38.770560 29090 catalog_manager.cc:4830] TS 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): tablet eb982ed7bcf74eb981c5199d87f9b4c4 (table rebalancer_extra_table_34 [id=ca1d67f8526844faaafcb2d933ff0572]) successfully deleted
W20240501 14:02:38.788697 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.805538 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.944684 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.959251 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.959596 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.986238 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:38.996171 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.034229 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.057219 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:39.071646 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:02:39.086942 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:39.097602 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:02:39.098291 32295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 14:02:39.103042 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:39.130404 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:39.141757 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:39.143960 29571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:02:39.145293 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.146310 32031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:02:39.147347 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.152676 31899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 14:02:39.154361 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.178117 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.186396 29311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:02:39.200531 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:39.205104 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:39.210386 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.226593 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.228089 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.234422 29180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:02:39.235425 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.253219 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.286451 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:39.300112 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:39.308537 14025 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:39.308763 14025 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:39.309623 14025 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:39.310874 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:02:39.311122 32096 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:39.318656 29571 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
W20240501 14:02:39.322183 29571 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:39.322371 29571 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:39.322671 14025 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:39.333117 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.335059 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.382467 32162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:02:39.413054 29245 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:39.415098 31964 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:39.415318 32360 raft_consensus.cc:1261] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:39.415825 14674 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:39.416131 14627 consensus_queue.cc:1035] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:39.418999 14674 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:39.419643 31965 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:39.420271 32360 raft_consensus.cc:2932] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:39.426456 29101 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 6 to 7, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:39.443832 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 578c3f4c5a7c4e2c962b64736195f0bc with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:39.453594 15060 tablet_replica.cc:330] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:39.453832 15060 raft_consensus.cc:2229] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:39.454084 15060 raft_consensus.cc:2258] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:39.454878 15060 ts_tablet_manager.cc:1875] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:39.462839 15060 ts_tablet_manager.cc:1888] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:02:39.463052 15060 log.cc:1196] T 578c3f4c5a7c4e2c962b64736195f0bc P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/578c3f4c5a7c4e2c962b64736195f0bc
I20240501 14:02:39.463898 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 578c3f4c5a7c4e2c962b64736195f0bc (table rebalance_test_table_1 [id=2f610d2751444a6786102f240f7d885f]) successfully deleted
W20240501 14:02:39.532182 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:39.538471 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:39.785449 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:39.794732 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:02:39.814678 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:39.841651 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:39.861016 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:39.888425 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:39.916121 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:39.921867 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:39.928383 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:39.958595 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:40.019277 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:40.103053 29245 consensus_queue.cc:237] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:40.105820 31965 raft_consensus.cc:1261] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:40.106233 32096 raft_consensus.cc:1261] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:02:40.106806 14627 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:40.107172 14674 consensus_queue.cc:1035] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:02:40.111003 14674 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:40.114652 31965 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:40.117658 29100 catalog_manager.cc:5484] T 3df490207f6248ce8a36687d71fedb85 P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New cstate: current_term: 1 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:40.119316 32096 raft_consensus.cc:2932] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b87d188552494b4590afc37a85b98c35 (127.22.33.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:40.143059 29356 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df490207f6248ce8a36687d71fedb85 with delete_type TABLET_DATA_TOMBSTONED (TS b87d188552494b4590afc37a85b98c35 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51134
I20240501 14:02:40.144176 15145 tablet_replica.cc:330] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: stopping tablet replica
I20240501 14:02:40.144385 15145 raft_consensus.cc:2229] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:40.144579 15145 raft_consensus.cc:2258] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:40.145227 15145 ts_tablet_manager.cc:1875] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:40.154305 15145 ts_tablet_manager.cc:1888] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:02:40.154428 15145 log.cc:1196] T 3df490207f6248ce8a36687d71fedb85 P b87d188552494b4590afc37a85b98c35: Deleting WAL directory at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/wals/3df490207f6248ce8a36687d71fedb85
I20240501 14:02:40.155129 29091 catalog_manager.cc:4830] TS b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): tablet 3df490207f6248ce8a36687d71fedb85 (table rebalancer_extra_table_56 [id=b7abdbf07069427caa9904f8b5e9d1de]) successfully deleted
W20240501 14:02:40.174408 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:40.254578 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:40.332566 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:40.336580 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:40.447305 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:40.484516 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:02:40.487438 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:40.504418 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:40.509469 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:40.517553 31944 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:40.528427 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:02:40.542572 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:40.558895 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:40.592130 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:40.616919 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:02:40.661108 29312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 95161.6us
W20240501 14:02:40.681314 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:40.682631 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:40.730237 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:40.759747 31964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d"
dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40242
I20240501 14:02:40.759986 31964 raft_consensus.cc:595] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:02:40.812536 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:40.897377 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:40.922348 14452 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:40.922514 14452 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:40.923228 14452 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:40.924799 29571 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:02:40.924852 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:02:40.925086 32096 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:40.925266 29571 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:40.925415 29571 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:40.929239 15116 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:40.966253 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.003141 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:02:41.104491 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:41.108847 14431 raft_consensus.cc:979] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7: : Instructing follower 6518b7a04854458bb2fb23edc8b6f7ba to start an election
I20240501 14:02:41.109058 14431 raft_consensus.cc:1067] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Signalling peer 6518b7a04854458bb2fb23edc8b6f7ba to start an election
I20240501 14:02:41.109776 29245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d"
dest_uuid: "6518b7a04854458bb2fb23edc8b6f7ba"
from {username='slave'} at 127.22.33.69:44499
I20240501 14:02:41.110024 29245 raft_consensus.cc:482] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:41.110138 29245 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:41.110755 29224 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:02:41.111920 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:41.123330 29245 raft_consensus.cc:504] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:41.124853 29245 leader_election.cc:288] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 election: Requested vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:41.125319 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e"
I20240501 14:02:41.125537 29635 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:41.128444 29635 raft_consensus.cc:2454] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 2.
I20240501 14:02:41.137167 31964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:02:41.137445 31964 raft_consensus.cc:3032] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:41.137563 31964 raft_consensus.cc:729] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
I20240501 14:02:41.141572 31964 consensus_queue.cc:260] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:41.142288 31964 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:41.151446 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.151691 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:41.152658 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.153569 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:41.155239 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.157191 31964 raft_consensus.cc:2454] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 2.
I20240501 14:02:41.189970 29181 leader_election.cc:302] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters:
W20240501 14:02:41.191090 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.192523 14627 raft_consensus.cc:2781] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:41.192708 14627 raft_consensus.cc:686] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [term 2 LEADER]: Becoming Leader. State: Replica: 6518b7a04854458bb2fb23edc8b6f7ba, State: Running, Role: LEADER
I20240501 14:02:41.193050 14627 consensus_queue.cc:237] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:41.196590 31944 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:02:41.224866 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.256690 29101 catalog_manager.cc:5484] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba reported cstate change: term changed from 1 to 2, leader changed from 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69) to 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65). New cstate: current_term: 2 leader_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } }
I20240501 14:02:41.266176 29374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c86d880c077411a99e6b9597adeeb0d" candidate_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b87d188552494b4590afc37a85b98c35"
I20240501 14:02:41.266407 29374 raft_consensus.cc:3037] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:41.269448 29374 raft_consensus.cc:2454] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6518b7a04854458bb2fb23edc8b6f7ba in term 2.
W20240501 14:02:41.293766 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.305192 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:41.329654 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.374732 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.396617 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.420076 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:41.448235 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.452245 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.506752 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.542225 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.566992 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.575731 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.607007 29377 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P b87d188552494b4590afc37a85b98c35 [term 2 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:41.610693 15272 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:41.616904 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.625334 29634 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P ecc8bf0965164f199fb0bfa3d7b47c0e [term 2 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:41.627596 15272 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:41.635743 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.648154 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.651695 31963 raft_consensus.cc:1261] T 6c86d880c077411a99e6b9597adeeb0d P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Refusing update from remote peer 6518b7a04854458bb2fb23edc8b6f7ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:41.652880 14627 consensus_queue.cc:1035] T 6c86d880c077411a99e6b9597adeeb0d P 6518b7a04854458bb2fb23edc8b6f7ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:41.666399 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.739251 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.745378 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.754401 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:41.811760 29224 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 14:02:41.850418 32162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 71890.4us
W20240501 14:02:41.852136 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:41.873863 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:41.875809 31944 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:02:41.878055 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.881556 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.883157 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:41.889940 29180 consensus_peers.cc:485] T 8b925d954c9143279ba69ae5c90d3266 P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.917047 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:41.917196 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:41.929200 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:41.961398 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:42.033356 29180 consensus_peers.cc:485] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.095477 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.274931 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.416337 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:42.417174 29224 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:02:42.438439 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:42.470873 31944 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:42.488737 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:42.487859 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:02:42.533238 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:02:42.559796 29180 consensus_peers.cc:485] T 10e1775ccab942ff857d22e89d99764d P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.570901 29571 consensus_peers.cc:485] T d2e5851626b54e1caf6105a70ccafc43 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:42.578531 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:42.633846 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.747596 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:42.771819 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:42.898025 31899 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.001173 29180 consensus_peers.cc:485] T e190940ae8b44ca2bf1ee496a9d624ba P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:43.021318 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:43.026973 32076 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:02:43.067626 29224 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 14:02:43.072088 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:43.094906 32207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:02:43.114248 31944 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:02:43.118906 32342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:02:43.149685 31899 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:43.180616 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:02:43.196331 29356 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:02:43.226289 29617 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 14:02:43.242749 31899 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.280042 29179 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:34947, user_credentials={real_user=slave}} blocked reactor thread for 61092.1us
W20240501 14:02:43.342839 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:02:43.433420 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:43.454118 32031 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.466426 29571 consensus_peers.cc:485] T 7e0fcc86017b4589b1a960c078e7ec57 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.504114 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.531550 31899 consensus_peers.cc:485] T ba3bd81448244b5bbdcd9eb8d23aee03 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.531834 29311 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:43.541407 15376 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:43.541613 15376 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:43.542456 15376 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:43.545503 32096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b72e2ac9df4426b071a7e5004e12e1" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
I20240501 14:02:43.545744 32096 raft_consensus.cc:2454] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8bf0965164f199fb0bfa3d7b47c0e in term 1.
W20240501 14:02:43.550433 29571 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:43.550850 29571 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
I20240501 14:02:43.550966 29571 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:43.551187 14025 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:43.611454 32031 consensus_peers.cc:485] T 0132590f8ed5423f983c53d9c19f95fe P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.708186 29311 consensus_peers.cc:485] T 67ed6fab7d9e46aa8faeb3cf784af0fb P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.723553 29571 consensus_peers.cc:485] T 5351f83631e64b7abe60575a6c285592 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.748497 32031 consensus_peers.cc:485] T fee3fe52d2ae462a823d3ac03f2ce8bc P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.753284 31899 consensus_peers.cc:485] T 3f354a7a3c4f4c61a62f2c3e0cbbf812 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.857223 29571 consensus_peers.cc:485] T 9e32147255254f2d8baff46c9b2d0e60 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.876328 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:43.917313 29180 consensus_peers.cc:485] T e67dc79a086048a8839ae386857652ea P 6518b7a04854458bb2fb23edc8b6f7ba -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.931460 32162 consensus_peers.cc:485] T b5c8bd2ddd7448bd8a2ccde39c08b763 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:43.937073 29571 consensus_peers.cc:485] T e513634d27c5467fb49c9717c43904cd P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskF0nCis/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 1
Maximum Replica Count | 43
Average Replica Count | 21.750000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.900000
stderr: W20240501 14:00:43.136811 32416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:43.137291 32416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:43.145401 32416 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
E20240501 14:00:43.772729 32520 ksck_remote.cc:634] unable to open table rebalance_test_table_2: Not found: the table does not exist: table_name: "rebalance_test_table_2"
I20240501 14:00:44.600379 32416 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:00:45.656800 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0f9c8a900a4a4a81864dc6d0d7731896: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.657153 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7db663aa0244455abdf483182accb2bc: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:45.657305 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 841e307cb41f45debe275139822553e6: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:45.657420 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:45.657513 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.657657 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7de1b8153edc41bf8ce503a2aa68891d: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.657826 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0d0478d77c2e42ceb6542048eb460da7: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:45.657934 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9d691283009410b92557f2186334223: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.658133 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e2b1173c6aad461d8f39f25a09d9d7a1: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:45.658218 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2f610d2751444a6786102f240f7d885f: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.658327 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 71f13072514a41e9be48406a0812b4ab: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:45.658435 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afecfebb80344e81b07b7d25a8ba0025: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.658565 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fb10ee2f136e4bb0a8471d277249bf0a: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:45.658769 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ce67b4011598474bad7e6571bf51640b: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.658880 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0da56e35536a4d32a2a1284a78195d3b: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:45.658962 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90e554b79da745d1ad57288ed7f66c03: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:45.659060 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2f610d2751444a6786102f240f7d885f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:45.659121 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 65cf90100af74431b5700c6a0046dbfa: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:45.659180 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 841e307cb41f45debe275139822553e6: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:45.659256 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:45.659323 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 71f13072514a41e9be48406a0812b4ab: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
I20240501 14:00:45.736052 32416 rebalancer_tool.cc:1074] tablet 9b76fa6c27c44d7588cd0fb133113bfe: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:00:45.833946 32416 rebalancer_tool.cc:1074] tablet 954bd347d6944d9cbc7587cb2435487e: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:00:45.975720 32416 rebalancer_tool.cc:1074] tablet 60527f81f34b45c199bf52bf0df53ee2: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:00:46.057817 32416 rebalancer_tool.cc:1074] tablet 49b72e2ac9df4426b071a7e5004e12e1: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:00:46.060194 32416 rebalancer_tool.cc:1084] tablet 8227f23eac39422eb5fa519d202a0d57: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move ignored: Not found: 8227f23eac39422eb5fa519d202a0d57: tablet not found
I20240501 14:00:46.060331 32416 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:00:47.085897 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fb10ee2f136e4bb0a8471d277249bf0a: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.086073 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e2b1173c6aad461d8f39f25a09d9d7a1: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.086216 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f4e5f6d094234e1fac8d9875ac375ed9: could not find any suitable replica to move from server 9903ed35bca144bcb74e9bf5102117e7 to server 522f4ff5e9084cbf8462196533c7d7aa
W20240501 14:00:47.086418 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7db663aa0244455abdf483182accb2bc: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:47.086509 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7de1b8153edc41bf8ce503a2aa68891d: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:47.086560 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fb10ee2f136e4bb0a8471d277249bf0a: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:47.086642 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 841e307cb41f45debe275139822553e6: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:47.086704 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0d0478d77c2e42ceb6542048eb460da7: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.086812 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ce67b4011598474bad7e6571bf51640b: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:47.086866 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:47.086937 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0da56e35536a4d32a2a1284a78195d3b: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:47.087014 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2f610d2751444a6786102f240f7d885f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.087096 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:47.087172 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afecfebb80344e81b07b7d25a8ba0025: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:47.087250 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fb10ee2f136e4bb0a8471d277249bf0a: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:47.087307 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9d691283009410b92557f2186334223: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:47.087453 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90e554b79da745d1ad57288ed7f66c03: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.087517 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:47.087604 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 0a697f3a1ee544daa7073814ab845d58
I20240501 14:00:47.176162 32416 rebalancer_tool.cc:1074] tablet a82cbc0696fe4a8a815eaf0538b5aad8: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:00:47.230147 32416 rebalancer_tool.cc:1074] tablet 414c148db46543b786bfd8b0faecfe96: '9903ed35bca144bcb74e9bf5102117e7' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:00:47.335069 32416 rebalancer_tool.cc:1074] tablet e513634d27c5467fb49c9717c43904cd: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:00:47.442255 32416 rebalancer_tool.cc:1074] tablet 5a8d8249c87f42faa9a7464369048090: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:00:48.051364 32416 rebalancer_tool.cc:1074] tablet 8b925d954c9143279ba69ae5c90d3266: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:00:48.375670 32416 rebalancer_tool.cc:1074] tablet 578c3f4c5a7c4e2c962b64736195f0bc: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:00:48.824517 32416 rebalancer_tool.cc:1074] tablet 6c86d880c077411a99e6b9597adeeb0d: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:00:49.109524 32416 rebalancer_tool.cc:1074] tablet 67ed6fab7d9e46aa8faeb3cf784af0fb: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:00:49.216930 32416 rebalancer_tool.cc:1074] tablet 5893a54b753f4d52a900d34140f6363d: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:00:49.296260 32416 rebalancer_tool.cc:1074] tablet a6e94a79c4514fe4a3ae8e41c7792dc9: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:00:49.302093 32416 rebalancer_tool.cc:1084] tablet b679ed4f63414ae1aefe8cd769b4b05c: '6518b7a04854458bb2fb23edc8b6f7ba' -> '0a697f3a1ee544daa7073814ab845d58' move ignored: Not found: b679ed4f63414ae1aefe8cd769b4b05c: tablet not found
I20240501 14:00:49.302242 32416 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:00:49.784934 881 meta_cache.cc:1330] Time spent looking up entry by ID: real 0.076s user 0.005s sys 0.000s
W20240501 14:00:49.785395 863 meta_cache.cc:1041] Time spent processing tablet locations response: real 0.116s user 0.001s sys 0.000s
W20240501 14:00:49.786135 863 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.22.33.126:35223, user_credentials={real_user=slave}} blocked reactor thread for 118750us
W20240501 14:00:50.592186 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:50.592438 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ce67b4011598474bad7e6571bf51640b: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:50.592604 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f4e5f6d094234e1fac8d9875ac375ed9: could not find any suitable replica to move from server 9903ed35bca144bcb74e9bf5102117e7 to server ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:00:50.592852 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e2b1173c6aad461d8f39f25a09d9d7a1: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:50.593032 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa51025b883748b986a6deefb4b7cda6: could not find any suitable replica to move from server 0a697f3a1ee544daa7073814ab845d58 to server 6518b7a04854458bb2fb23edc8b6f7ba
W20240501 14:00:50.593127 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:50.593196 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9d691283009410b92557f2186334223: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 0a697f3a1ee544daa7073814ab845d58
W20240501 14:00:50.593291 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 841e307cb41f45debe275139822553e6: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:50.593386 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:00:50.593536 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90e554b79da745d1ad57288ed7f66c03: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:50.593683 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2f610d2751444a6786102f240f7d885f: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:50.593819 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7de1b8153edc41bf8ce503a2aa68891d: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:50.593932 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0da56e35536a4d32a2a1284a78195d3b: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:00:50.594043 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c83656dad3542fab20e5318a759b9bc: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:00:50.594156 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4774d7871124ad69e2971b30257f095: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:00:50.658129 32416 rebalancer_tool.cc:1074] tablet e5e2a1f3418c477aa65b77bfa171a5a1: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:00:50.719360 32416 rebalancer_tool.cc:1074] tablet fee3fe52d2ae462a823d3ac03f2ce8bc: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:00:50.767890 32416 rebalancer_tool.cc:1074] tablet 70ffd2100f104b339bfdc2d272ba3729: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:00:50.855039 32416 rebalancer_tool.cc:1074] tablet 820be6be18af4781ace626f239379cff: '0a697f3a1ee544daa7073814ab845d58' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:00:51.253005 32416 rebalancer_tool.cc:1074] tablet 1a412c988fd946d7b7d51606be01d1fa: '6edd6a3430d44c38a19c2804e555059f' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:00:57.122431 32416 rebalancer_tool.cc:1123] tablet 9b76fa6c27c44d7588cd0fb133113bfe: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move is abandoned: Not found: 9b76fa6c27c44d7588cd0fb133113bfe: tablet not found
W20240501 14:01:04.875834 2771 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.ServerClock -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 82341.2us
I20240501 14:01:06.315732 32416 rebalancer_tool.cc:1123] tablet 954bd347d6944d9cbc7587cb2435487e: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move is abandoned: Not found: 954bd347d6944d9cbc7587cb2435487e: tablet not found
W20240501 14:01:06.994381 3086 meta_cache.cc:1261] Time spent looking up entry by key: real 0.086s user 0.001s sys 0.000s
I20240501 14:01:11.695622 32416 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:01:12.381666 3584 meta_cache.cc:1041] Time spent processing tablet locations response: real 0.075s user 0.000s sys 0.000s
W20240501 14:01:12.384824 3584 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.22.33.126:35223, user_credentials={real_user=slave}} blocked reactor thread for 78470.6us
W20240501 14:01:13.326320 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0d0b90f25384d879d18bbfc7e4b20dd: could not find any suitable replica to move from server 9903ed35bca144bcb74e9bf5102117e7 to server ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:01:13.326844 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:01:13.327001 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ce67b4011598474bad7e6571bf51640b: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:01:13.327157 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0457d1d1f49c42b2b7a2d3bdd9bbd07b: could not find any suitable replica to move from server 6edd6a3430d44c38a19c2804e555059f to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:01:13.327704 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca1d67f8526844faaafcb2d933ff0572: could not find any suitable replica to move from server 0a697f3a1ee544daa7073814ab845d58 to server ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:01:13.327860 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 47c80c186884434d8ade8ed82e71ffcb: could not find any suitable replica to move from server 0a697f3a1ee544daa7073814ab845d58 to server 522f4ff5e9084cbf8462196533c7d7aa
W20240501 14:01:13.328152 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7de1b8153edc41bf8ce503a2aa68891d: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:01:13.328255 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e2b8a4427e44b4184a0cfcc7047d71a: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:01:13.328403 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 841e307cb41f45debe275139822553e6: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:01:13.328516 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9d691283009410b92557f2186334223: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 6edd6a3430d44c38a19c2804e555059f
W20240501 14:01:13.328832 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e2b1173c6aad461d8f39f25a09d9d7a1: could not find any suitable replica to move from server ecc8bf0965164f199fb0bfa3d7b47c0e to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:01:13.329041 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2f610d2751444a6786102f240f7d885f: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:01:13.329263 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90e554b79da745d1ad57288ed7f66c03: could not find any suitable replica to move from server 522f4ff5e9084cbf8462196533c7d7aa to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:01:13.329443 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c437ad9a367549aebe99d225bb56580f: could not find any suitable replica to move from server 9903ed35bca144bcb74e9bf5102117e7 to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:01:13.329581 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 855664ff47f14f92a553681cf6c9e7d6: could not find any suitable replica to move from server 6a446ac802a7457d9f27076781f21d79 to server 522f4ff5e9084cbf8462196533c7d7aa
W20240501 14:01:13.329741 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d03642db77f47678c8ba46b8f46c89d: could not find any suitable replica to move from server 0a697f3a1ee544daa7073814ab845d58 to server 9903ed35bca144bcb74e9bf5102117e7
W20240501 14:01:13.329960 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28fe00a56c2e4f1a88166fb5d1cf5628: could not find any suitable replica to move from server 6edd6a3430d44c38a19c2804e555059f to server b87d188552494b4590afc37a85b98c35
W20240501 14:01:13.330099 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa51025b883748b986a6deefb4b7cda6: could not find any suitable replica to move from server 6a446ac802a7457d9f27076781f21d79 to server 6518b7a04854458bb2fb23edc8b6f7ba
W20240501 14:01:13.330407 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0da56e35536a4d32a2a1284a78195d3b: could not find any suitable replica to move from server 6518b7a04854458bb2fb23edc8b6f7ba to server 6a446ac802a7457d9f27076781f21d79
W20240501 14:01:13.330665 32416 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35020b4bdfd14884929b6c060344c01f: could not find any suitable replica to move from server b87d188552494b4590afc37a85b98c35 to server ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:01:13.386078 32416 rebalancer_tool.cc:1074] tablet c696bdff78504c0c93f39a7e07a40dcd: '6518b7a04854458bb2fb23edc8b6f7ba' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:13.479192 32416 rebalancer_tool.cc:1074] tablet 798c9c25344d4beb9a079fb776044a17: '6a446ac802a7457d9f27076781f21d79' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:01:15.564558 32416 rebalancer_tool.cc:1136] tablet 70ffd2100f104b339bfdc2d272ba3729: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:16.243664 32416 rebalancer_tool.cc:1136] tablet 49b72e2ac9df4426b071a7e5004e12e1: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:16.255431 32416 rebalancer_tool.cc:1136] tablet 60527f81f34b45c199bf52bf0df53ee2: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:01:16.265045 32416 rebalancer_tool.cc:1136] tablet 67ed6fab7d9e46aa8faeb3cf784af0fb: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:16.275703 32416 rebalancer_tool.cc:1136] tablet a82cbc0696fe4a8a815eaf0538b5aad8: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:16.285624 32416 rebalancer_tool.cc:1136] tablet e513634d27c5467fb49c9717c43904cd: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:16.296758 32416 rebalancer_tool.cc:1136] tablet 1a412c988fd946d7b7d51606be01d1fa: '6edd6a3430d44c38a19c2804e555059f' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:16.305228 32416 rebalancer_tool.cc:1136] tablet 8b925d954c9143279ba69ae5c90d3266: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:01:16.313709 32416 rebalancer_tool.cc:1136] tablet fee3fe52d2ae462a823d3ac03f2ce8bc: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:01:16.321985 32416 rebalancer_tool.cc:1136] tablet 578c3f4c5a7c4e2c962b64736195f0bc: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:16.331866 32416 rebalancer_tool.cc:1136] tablet 6c86d880c077411a99e6b9597adeeb0d: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:16.339680 32416 rebalancer_tool.cc:1136] tablet 5893a54b753f4d52a900d34140f6363d: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:01:16.349573 32416 rebalancer_tool.cc:1136] tablet 5a8d8249c87f42faa9a7464369048090: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:16.359483 32416 rebalancer_tool.cc:1136] tablet a6e94a79c4514fe4a3ae8e41c7792dc9: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:16.368482 32416 rebalancer_tool.cc:1136] tablet e5e2a1f3418c477aa65b77bfa171a5a1: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:16.411603 32416 rebalancer_tool.cc:1074] tablet e83b7b4decde4e459e49de9eb05b8c0f: '522f4ff5e9084cbf8462196533c7d7aa' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:16.546731 32416 rebalancer_tool.cc:1074] tablet 59c54188ad9e40dd925b50a250dcea04: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:01:16.605926 32416 rebalancer_tool.cc:1074] tablet ba20c01d741a414196972ced1601baf5: '6518b7a04854458bb2fb23edc8b6f7ba' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:16.656728 32416 rebalancer_tool.cc:1074] tablet 241a2fdfe954476ba9bd2cfeb4b931ac: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:01:16.717922 32416 rebalancer_tool.cc:1074] tablet 62c84a393bd047d985927417f82bec7d: '9903ed35bca144bcb74e9bf5102117e7' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:16.778399 32416 rebalancer_tool.cc:1074] tablet 8427dee24ad04247b23b24570c73c92a: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:01:16.861932 32416 rebalancer_tool.cc:1074] tablet e190940ae8b44ca2bf1ee496a9d624ba: '0a697f3a1ee544daa7073814ab845d58' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:16.941841 32416 rebalancer_tool.cc:1074] tablet ca0a888dc37d44c69dea73f9ffd556f6: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:01:17.299551 32416 rebalancer_tool.cc:1074] tablet 9c7c0ef199394de490b790a26532df3b: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:17.445926 32416 rebalancer_tool.cc:1074] tablet eb3c16421adb477e821495e785849cc4: '9903ed35bca144bcb74e9bf5102117e7' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:17.638063 32416 rebalancer_tool.cc:1074] tablet 6fbb60619c1a4d8e8cd6266596a0614d: '6a446ac802a7457d9f27076781f21d79' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:01:17.724929 32416 rebalancer_tool.cc:1074] tablet 34d3876037334884a4a36bc192956cf4: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:17.804895 32416 rebalancer_tool.cc:1074] tablet 0ff2fc8d230a45daa4e0c8ba4fec422e: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:17.910843 32416 rebalancer_tool.cc:1074] tablet eb982ed7bcf74eb981c5199d87f9b4c4: '9903ed35bca144bcb74e9bf5102117e7' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:18.059206 32416 rebalancer_tool.cc:1074] tablet 500b9c9bdfa649bd883d6ef2d6e541fc: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:01:18.120940 32416 rebalancer_tool.cc:1074] tablet bfbf43ae81ae45ba8d5abd345d18ac8a: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
W20240501 14:01:25.517555 5416 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 50250.3us
I20240501 14:01:30.566151 32416 rebalancer_tool.cc:1136] tablet c696bdff78504c0c93f39a7e07a40dcd: '6518b7a04854458bb2fb23edc8b6f7ba' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:01:31.181702 32416 rebalancer_tool.cc:1136] tablet 820be6be18af4781ace626f239379cff: '0a697f3a1ee544daa7073814ab845d58' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:01:31.199213 32416 rebalancer_tool.cc:1136] tablet 414c148db46543b786bfd8b0faecfe96: '9903ed35bca144bcb74e9bf5102117e7' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:31.263193 32416 rebalancer_tool.cc:1074] tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d: '6518b7a04854458bb2fb23edc8b6f7ba' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:01:31.329895 32416 rebalancer_tool.cc:1074] tablet 0efddc01769f43e0b6f44d40ce6fa562: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:32.104454 32416 rebalancer_tool.cc:1136] tablet eb982ed7bcf74eb981c5199d87f9b4c4: '9903ed35bca144bcb74e9bf5102117e7' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:32.120880 32416 rebalancer_tool.cc:1136] tablet eb3c16421adb477e821495e785849cc4: '9903ed35bca144bcb74e9bf5102117e7' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:32.837839 32416 rebalancer_tool.cc:1136] tablet 9c7c0ef199394de490b790a26532df3b: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:33.505219 32416 rebalancer_tool.cc:1136] tablet e190940ae8b44ca2bf1ee496a9d624ba: '0a697f3a1ee544daa7073814ab845d58' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:33.551208 32416 rebalancer_tool.cc:1136] tablet 8427dee24ad04247b23b24570c73c92a: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:33.567081 32416 rebalancer_tool.cc:1136] tablet 62c84a393bd047d985927417f82bec7d: '9903ed35bca144bcb74e9bf5102117e7' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:33.575783 32416 rebalancer_tool.cc:1136] tablet ca0a888dc37d44c69dea73f9ffd556f6: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:33.588758 32416 rebalancer_tool.cc:1136] tablet 241a2fdfe954476ba9bd2cfeb4b931ac: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:35.019294 32416 rebalancer_tool.cc:1136] tablet 0ff2fc8d230a45daa4e0c8ba4fec422e: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:01:35.031488 32416 rebalancer_tool.cc:1136] tablet 59c54188ad9e40dd925b50a250dcea04: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:35.042507 32416 rebalancer_tool.cc:1136] tablet e83b7b4decde4e459e49de9eb05b8c0f: '522f4ff5e9084cbf8462196533c7d7aa' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:35.062860 32416 rebalancer_tool.cc:1136] tablet 798c9c25344d4beb9a079fb776044a17: '6a446ac802a7457d9f27076781f21d79' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:35.074682 32416 rebalancer_tool.cc:1136] tablet bfbf43ae81ae45ba8d5abd345d18ac8a: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:35.085773 32416 rebalancer_tool.cc:1136] tablet 34d3876037334884a4a36bc192956cf4: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:35.143206 32416 rebalancer_tool.cc:1074] tablet 25a6e3129a2f4bf89a36cdca5eaeb376: '6edd6a3430d44c38a19c2804e555059f' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:35.231076 32416 rebalancer_tool.cc:1074] tablet 36b0222ebb6d4246ab8551d5579b1760: '9903ed35bca144bcb74e9bf5102117e7' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:35.298671 32416 rebalancer_tool.cc:1074] tablet d2e5851626b54e1caf6105a70ccafc43: 'b87d188552494b4590afc37a85b98c35' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:01:35.358918 32416 rebalancer_tool.cc:1074] tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:35.411818 32416 rebalancer_tool.cc:1074] tablet c6973301bdd94c2b8f98e718715ff752: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:35.519855 32416 rebalancer_tool.cc:1074] tablet e5ed5a5797424f0f88c2b3c711468e6b: '9903ed35bca144bcb74e9bf5102117e7' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:35.606122 32416 rebalancer_tool.cc:1074] tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d: 'b87d188552494b4590afc37a85b98c35' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:01:35.687906 32416 rebalancer_tool.cc:1074] tablet 5c0fc97541db4467b5089bbeaf9bfca0: '6edd6a3430d44c38a19c2804e555059f' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:35.769670 32416 rebalancer_tool.cc:1074] tablet 71933eccec5f4fdc8b1df7111f941777: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:35.883246 32416 rebalancer_tool.cc:1074] tablet 9e32147255254f2d8baff46c9b2d0e60: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:01:36.113233 32416 rebalancer_tool.cc:1074] tablet fe99c587804f47c3b58ec3ddbab8a371: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:36.321568 32416 rebalancer_tool.cc:1074] tablet e67dc79a086048a8839ae386857652ea: '6a446ac802a7457d9f27076781f21d79' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:36.493722 32416 rebalancer_tool.cc:1074] tablet ab3c1d804ba74db794bc0abe68bc9f96: '0a697f3a1ee544daa7073814ab845d58' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:36.612450 32416 rebalancer_tool.cc:1074] tablet b5aa71ecef4d419eb36c7777790e6f08: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:47.049640 32416 rebalancer_tool.cc:1136] tablet 500b9c9bdfa649bd883d6ef2d6e541fc: '522f4ff5e9084cbf8462196533c7d7aa' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:47.061003 32416 rebalancer_tool.cc:1136] tablet 0efddc01769f43e0b6f44d40ce6fa562: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:47.070490 32416 rebalancer_tool.cc:1136] tablet 6fbb60619c1a4d8e8cd6266596a0614d: '6a446ac802a7457d9f27076781f21d79' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:01:47.081146 32416 rebalancer_tool.cc:1136] tablet 3f3d8b11a9bd4f69a855aa087e7cbb5d: '6518b7a04854458bb2fb23edc8b6f7ba' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:01:47.090934 32416 rebalancer_tool.cc:1136] tablet ba20c01d741a414196972ced1601baf5: '6518b7a04854458bb2fb23edc8b6f7ba' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:47.151894 32416 rebalancer_tool.cc:1074] tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:47.282112 32416 rebalancer_tool.cc:1074] tablet 192a6cc308da4a448f03717b6a439094: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:01:47.354207 32416 rebalancer_tool.cc:1074] tablet fa4a1cdf1c214e929b1d81ae3df585d9: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:47.418752 32416 rebalancer_tool.cc:1074] tablet a87b1de51c4c4f909c27c27aa745fb44: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:01:47.545094 32416 rebalancer_tool.cc:1074] tablet 9d39eab213cc464ba90149276cfeeac8: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
W20240501 14:01:48.162817 8490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.33.65:43201, user_credentials={real_user=slave}} blocked reactor thread for 48749.3us
W20240501 14:01:48.294579 8491 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.22.33.72:37615, user_credentials={real_user=slave}} blocked reactor thread for 88356.3us
I20240501 14:01:51.888674 32416 rebalancer_tool.cc:1136] tablet fe99c587804f47c3b58ec3ddbab8a371: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:01:51.902899 32416 rebalancer_tool.cc:1136] tablet 9e32147255254f2d8baff46c9b2d0e60: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:01:51.913170 32416 rebalancer_tool.cc:1136] tablet 5c0fc97541db4467b5089bbeaf9bfca0: '6edd6a3430d44c38a19c2804e555059f' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:51.927630 32416 rebalancer_tool.cc:1136] tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d: 'b87d188552494b4590afc37a85b98c35' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:01:51.939666 32416 rebalancer_tool.cc:1136] tablet 71933eccec5f4fdc8b1df7111f941777: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:51.951643 32416 rebalancer_tool.cc:1136] tablet e5ed5a5797424f0f88c2b3c711468e6b: '9903ed35bca144bcb74e9bf5102117e7' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:01:51.962567 32416 rebalancer_tool.cc:1136] tablet ab3c1d804ba74db794bc0abe68bc9f96: '0a697f3a1ee544daa7073814ab845d58' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:51.973330 32416 rebalancer_tool.cc:1136] tablet c6973301bdd94c2b8f98e718715ff752: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:01:51.985543 32416 rebalancer_tool.cc:1136] tablet e67dc79a086048a8839ae386857652ea: '6a446ac802a7457d9f27076781f21d79' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:51.997834 32416 rebalancer_tool.cc:1136] tablet 7ec9bb1d59ac4b11a1f9b0d44b1dc69b: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:01:52.763470 32416 rebalancer_tool.cc:1136] tablet d2e5851626b54e1caf6105a70ccafc43: 'b87d188552494b4590afc37a85b98c35' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:01:52.777474 32416 rebalancer_tool.cc:1136] tablet 36b0222ebb6d4246ab8551d5579b1760: '9903ed35bca144bcb74e9bf5102117e7' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:01:52.798694 32416 rebalancer_tool.cc:1136] tablet 25a6e3129a2f4bf89a36cdca5eaeb376: '6edd6a3430d44c38a19c2804e555059f' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:01:52.894114 32416 rebalancer_tool.cc:1074] tablet 1a025068d40c4b388421c5e6930847c2: '0a697f3a1ee544daa7073814ab845d58' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:01:52.992841 32416 rebalancer_tool.cc:1074] tablet 1eb574dc421745618ba2a11f0b3f9110: 'b87d188552494b4590afc37a85b98c35' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:53.063825 32416 rebalancer_tool.cc:1074] tablet 77ce342cbfad4419b3e2e8d3a52c4628: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:01:53.118090 32416 rebalancer_tool.cc:1074] tablet 7fce4b1a15bc48e9b8042869d5c18d16: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:01:53.241446 32416 rebalancer_tool.cc:1074] tablet 7e0fcc86017b4589b1a960c078e7ec57: '6edd6a3430d44c38a19c2804e555059f' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:53.312583 32416 rebalancer_tool.cc:1074] tablet 815ce94772f2454ea755d33dd04a87d2: '6a446ac802a7457d9f27076781f21d79' -> 'b87d188552494b4590afc37a85b98c35' move scheduled
I20240501 14:01:53.380342 32416 rebalancer_tool.cc:1074] tablet ef4b00f102874e17830f2e340255a11f: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:01:53.454044 32416 rebalancer_tool.cc:1074] tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:01:53.571715 32416 rebalancer_tool.cc:1074] tablet f345c7a43d2648b4b4c47e726c4d4c51: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:01:53.689577 32416 rebalancer_tool.cc:1074] tablet 54bdc21d1dc644e7ac44f5b9ad39ab32: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:01:53.756440 32416 rebalancer_tool.cc:1074] tablet 21c02994dfef4a10a87b1f47d3ab6e3b: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:02.287037 32416 rebalancer_tool.cc:1136] tablet fa4a1cdf1c214e929b1d81ae3df585d9: '0a697f3a1ee544daa7073814ab845d58' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:02:02.296674 32416 rebalancer_tool.cc:1136] tablet 9d39eab213cc464ba90149276cfeeac8: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:02:02.305626 32416 rebalancer_tool.cc:1136] tablet 192a6cc308da4a448f03717b6a439094: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:02:04.329172 32416 rebalancer_tool.cc:1136] tablet 356722d5ec9d4d3d8ba08b4e7c2b2b11: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:02:04.378067 32416 rebalancer_tool.cc:1074] tablet b12b60d5bf304c28ae2041629b8d5bdd: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:04.439579 32416 rebalancer_tool.cc:1074] tablet c23d0b871aff4ce3850e45ae31097c03: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:04.566092 32416 rebalancer_tool.cc:1074] tablet cd7a7501266946dca6bc838d10524c4a: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:04.637001 32416 rebalancer_tool.cc:1074] tablet 768543d354744b44acdebb9a95a1579b: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:06.263397 32416 rebalancer_tool.cc:1136] tablet 3d8ce8f6f27b4ab19ff5a1bb972ca202: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:02:07.672720 32416 rebalancer_tool.cc:1136] tablet ef4b00f102874e17830f2e340255a11f: '6a446ac802a7457d9f27076781f21d79' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:02:07.682476 32416 rebalancer_tool.cc:1136] tablet 7e0fcc86017b4589b1a960c078e7ec57: '6edd6a3430d44c38a19c2804e555059f' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:07.692060 32416 rebalancer_tool.cc:1136] tablet 7fce4b1a15bc48e9b8042869d5c18d16: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:08.434190 32416 rebalancer_tool.cc:1136] tablet 1eb574dc421745618ba2a11f0b3f9110: 'b87d188552494b4590afc37a85b98c35' -> '6518b7a04854458bb2fb23edc8b6f7ba' move completed: OK
I20240501 14:02:09.096305 32416 rebalancer_tool.cc:1136] tablet 1a025068d40c4b388421c5e6930847c2: '0a697f3a1ee544daa7073814ab845d58' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:02:09.713275 32416 rebalancer_tool.cc:1136] tablet a87b1de51c4c4f909c27c27aa745fb44: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6a446ac802a7457d9f27076781f21d79' move completed: OK
I20240501 14:02:09.737185 32416 rebalancer_tool.cc:1136] tablet b5aa71ecef4d419eb36c7777790e6f08: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:09.748684 32416 rebalancer_tool.cc:1136] tablet 21c02994dfef4a10a87b1f47d3ab6e3b: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:09.757980 32416 rebalancer_tool.cc:1136] tablet 815ce94772f2454ea755d33dd04a87d2: '6a446ac802a7457d9f27076781f21d79' -> 'b87d188552494b4590afc37a85b98c35' move completed: OK
I20240501 14:02:09.802781 32416 rebalancer_tool.cc:1074] tablet b0d8239317634874b1ffb1ba239bb9ad: '6edd6a3430d44c38a19c2804e555059f' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:09.899912 32416 rebalancer_tool.cc:1074] tablet db862a636e5a427085a2fa31f6fe8050: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:09.960956 32416 rebalancer_tool.cc:1074] tablet 8a1690e3a76644859a0711e305b8f946: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:10.033706 32416 rebalancer_tool.cc:1074] tablet 68f3644410a24fc1a9457ff46a10c180: '0a697f3a1ee544daa7073814ab845d58' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:12.895481 32416 rebalancer_tool.cc:1136] tablet 54bdc21d1dc644e7ac44f5b9ad39ab32: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:12.913375 32416 rebalancer_tool.cc:1136] tablet b12b60d5bf304c28ae2041629b8d5bdd: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:12.925724 32416 rebalancer_tool.cc:1136] tablet f345c7a43d2648b4b4c47e726c4d4c51: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:13.616459 32416 rebalancer_tool.cc:1136] tablet 77ce342cbfad4419b3e2e8d3a52c4628: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move completed: OK
I20240501 14:02:13.631525 32416 rebalancer_tool.cc:1136] tablet cd7a7501266946dca6bc838d10524c4a: '9903ed35bca144bcb74e9bf5102117e7' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:13.644712 32416 rebalancer_tool.cc:1136] tablet 768543d354744b44acdebb9a95a1579b: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:13.696564 32416 rebalancer_tool.cc:1074] tablet c2c63489706e4d068adde1872f451ad2: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:02:13.749174 32416 rebalancer_tool.cc:1074] tablet 9c7cd18adcd844f4be7ddd343cda3df4: '522f4ff5e9084cbf8462196533c7d7aa' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:13.821755 32416 rebalancer_tool.cc:1074] tablet ba3bd81448244b5bbdcd9eb8d23aee03: '0a697f3a1ee544daa7073814ab845d58' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:14.051733 32416 rebalancer_tool.cc:1074] tablet 0132590f8ed5423f983c53d9c19f95fe: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:19.169785 32416 rebalancer_tool.cc:1136] tablet db862a636e5a427085a2fa31f6fe8050: '6a446ac802a7457d9f27076781f21d79' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:19.214805 32416 rebalancer_tool.cc:1136] tablet b0d8239317634874b1ffb1ba239bb9ad: '6edd6a3430d44c38a19c2804e555059f' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:19.226639 32416 rebalancer_tool.cc:1136] tablet c23d0b871aff4ce3850e45ae31097c03: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:19.252791 32416 rebalancer_tool.cc:1136] tablet 68f3644410a24fc1a9457ff46a10c180: '0a697f3a1ee544daa7073814ab845d58' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:19.262902 32416 rebalancer_tool.cc:1136] tablet 0132590f8ed5423f983c53d9c19f95fe: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:19.272732 32416 rebalancer_tool.cc:1136] tablet ba3bd81448244b5bbdcd9eb8d23aee03: '0a697f3a1ee544daa7073814ab845d58' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:19.282639 32416 rebalancer_tool.cc:1136] tablet 9c7cd18adcd844f4be7ddd343cda3df4: '522f4ff5e9084cbf8462196533c7d7aa' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move completed: OK
I20240501 14:02:19.292079 32416 rebalancer_tool.cc:1136] tablet c2c63489706e4d068adde1872f451ad2: '522f4ff5e9084cbf8462196533c7d7aa' -> '6edd6a3430d44c38a19c2804e555059f' move completed: OK
I20240501 14:02:19.317584 32416 rebalancer_tool.cc:1136] tablet 8a1690e3a76644859a0711e305b8f946: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:19.426894 32416 rebalancer_tool.cc:1074] tablet ad7af304700545b2a28249f9f046cab6: '0a697f3a1ee544daa7073814ab845d58' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:19.642364 32416 rebalancer_tool.cc:1074] tablet 10e1775ccab942ff857d22e89d99764d: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:19.709560 32416 rebalancer_tool.cc:1074] tablet 83ca18280bbb49058e5aa80a3832b666: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:19.762809 32416 rebalancer_tool.cc:1074] tablet c8020272a8b14dea8a6a82802ea4e10a: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:19.811350 32416 rebalancer_tool.cc:1074] tablet 0e33e4f9b1204797aeff0bca36c93911: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:25.661751 32416 rebalancer_tool.cc:1136] tablet c8020272a8b14dea8a6a82802ea4e10a: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:25.670838 32416 rebalancer_tool.cc:1136] tablet 83ca18280bbb49058e5aa80a3832b666: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:26.666229 32416 rebalancer_tool.cc:1136] tablet ad7af304700545b2a28249f9f046cab6: '0a697f3a1ee544daa7073814ab845d58' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:26.681392 32416 rebalancer_tool.cc:1136] tablet 0e33e4f9b1204797aeff0bca36c93911: '522f4ff5e9084cbf8462196533c7d7aa' -> '0a697f3a1ee544daa7073814ab845d58' move completed: OK
I20240501 14:02:26.692799 32416 rebalancer_tool.cc:1136] tablet 10e1775ccab942ff857d22e89d99764d: '6edd6a3430d44c38a19c2804e555059f' -> '522f4ff5e9084cbf8462196533c7d7aa' move completed: OK
I20240501 14:02:27.413554 32416 rebalancer_tool.cc:1074] tablet 3ec202bb41a44e05939f1ecdf33d8c38: '6518b7a04854458bb2fb23edc8b6f7ba' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:27.487187 32416 rebalancer_tool.cc:1074] tablet 578c3f4c5a7c4e2c962b64736195f0bc: 'b87d188552494b4590afc37a85b98c35' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:02:27.600864 32416 rebalancer_tool.cc:1074] tablet 54bdc21d1dc644e7ac44f5b9ad39ab32: '6a446ac802a7457d9f27076781f21d79' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:02:27.653853 32416 rebalancer_tool.cc:1074] tablet 768543d354744b44acdebb9a95a1579b: '0a697f3a1ee544daa7073814ab845d58' -> '6518b7a04854458bb2fb23edc8b6f7ba' move scheduled
I20240501 14:02:27.727095 32416 rebalancer_tool.cc:1074] tablet 6c86d880c077411a99e6b9597adeeb0d: '9903ed35bca144bcb74e9bf5102117e7' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:27.779850 32416 rebalancer_tool.cc:1074] tablet fa4a1cdf1c214e929b1d81ae3df585d9: '6a446ac802a7457d9f27076781f21d79' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:27.839483 32416 rebalancer_tool.cc:1074] tablet 500b9c9bdfa649bd883d6ef2d6e541fc: 'b87d188552494b4590afc37a85b98c35' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:02:27.957080 32416 rebalancer_tool.cc:1074] tablet 413a79f6a61a49d9a49a5a52cec4c00f: '0a697f3a1ee544daa7073814ab845d58' -> '522f4ff5e9084cbf8462196533c7d7aa' move scheduled
I20240501 14:02:28.119550 32416 rebalancer_tool.cc:1074] tablet 49b72e2ac9df4426b071a7e5004e12e1: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:02:28.345371 32416 rebalancer_tool.cc:1074] tablet 71933eccec5f4fdc8b1df7111f941777: '6518b7a04854458bb2fb23edc8b6f7ba' -> '6edd6a3430d44c38a19c2804e555059f' move scheduled
I20240501 14:02:28.497153 32416 rebalancer_tool.cc:1074] tablet fe99c587804f47c3b58ec3ddbab8a371: '9903ed35bca144bcb74e9bf5102117e7' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:02:28.740079 32416 rebalancer_tool.cc:1074] tablet 3df490207f6248ce8a36687d71fedb85: 'b87d188552494b4590afc37a85b98c35' -> '0a697f3a1ee544daa7073814ab845d58' move scheduled
I20240501 14:02:29.026705 32416 rebalancer_tool.cc:1074] tablet eb982ed7bcf74eb981c5199d87f9b4c4: '6edd6a3430d44c38a19c2804e555059f' -> 'ecc8bf0965164f199fb0bfa3d7b47c0e' move scheduled
I20240501 14:02:30.336300 32416 rebalancer_tool.cc:1074] tablet 4a38ce68ec4d4ba3a813bdb0ec314d4d: '522f4ff5e9084cbf8462196533c7d7aa' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:02:30.441797 32416 rebalancer_tool.cc:1074] tablet 192a6cc308da4a448f03717b6a439094: 'b87d188552494b4590afc37a85b98c35' -> '6a446ac802a7457d9f27076781f21d79' move scheduled
I20240501 14:02:30.638612 32416 rebalancer_tool.cc:1074] tablet c6973301bdd94c2b8f98e718715ff752: 'b87d188552494b4590afc37a85b98c35' -> '9903ed35bca144bcb74e9bf5102117e7' move scheduled
I20240501 14:02:34.280308 32416 rebalancer_tool.cc:1123] tablet 49b72e2ac9df4426b071a7e5004e12e1: 'ecc8bf0965164f199fb0bfa3d7b47c0e' -> '6edd6a3430d44c38a19c2804e555059f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:42.671836 32416 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): unacceptable health status UNAVAILABLE
I20240501 14:02:43.955211 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29164
I20240501 14:02:43.964246 29286 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:44.021534 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.026239 31899 consensus_peers.cc:485] T c3f0609897bd4890952f7535cfe737f2 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.042732 31899 consensus_peers.cc:485] T e2d4f5ec80e64aa6aceb42a0748af62b P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.064308 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.065040 29571 consensus_peers.cc:485] T ba20c01d741a414196972ced1601baf5 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.102890 31899 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.128413 32162 consensus_peers.cc:485] T 4d374fd761284e32913896ac5a22ab63 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.135782 29311 consensus_peers.cc:485] T a87b1de51c4c4f909c27c27aa745fb44 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.163301 29571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:02:44.176689 32031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 14:02:44.200565 29571 consensus_peers.cc:485] T 8a1690e3a76644859a0711e305b8f946 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.220678 31899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240501 14:02:44.235249 32295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:02:44.236033 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.262068 29311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:02:44.282629 32031 consensus_peers.cc:485] T 4a38ce68ec4d4ba3a813bdb0ec314d4d P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.317637 32295 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:44.373085 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29164
W20240501 14:02:44.386407 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:44.399045 32162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:02:44.407074 32295 consensus_peers.cc:485] T cd7a7501266946dca6bc838d10524c4a P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.440179 31898 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.440374 29310 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.440429 29570 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.440730 32294 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.441287 32161 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.441316 32030 connection.cc:535] client connection to 127.22.33.65:43201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.441820 29310 consensus_peers.cc:485] T ef4b00f102874e17830f2e340255a11f P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.441917 29312 connection.cc:535] server connection from 127.22.33.65:35753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.441927 29310 consensus_peers.cc:485] T 7da6bff08fa14cf6b4894f58e248fb59 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442092 29310 consensus_peers.cc:485] T e41986f1ae7648a5bfe7760c069af672 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442186 29310 consensus_peers.cc:485] T c2c63489706e4d068adde1872f451ad2 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442255 29310 consensus_peers.cc:485] T a6e94a79c4514fe4a3ae8e41c7792dc9 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442313 29310 consensus_peers.cc:485] T eb3c16421adb477e821495e785849cc4 P b87d188552494b4590afc37a85b98c35 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:44.442838 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29295
W20240501 14:02:44.442864 32161 consensus_peers.cc:485] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442898 29570 consensus_peers.cc:485] T 798c9c25344d4beb9a079fb776044a17 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.443002 29570 consensus_peers.cc:485] T 768543d354744b44acdebb9a95a1579b P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.442999 32161 consensus_peers.cc:485] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.443087 29570 consensus_peers.cc:485] T ab3c1d804ba74db794bc0abe68bc9f96 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.443161 29570 consensus_peers.cc:485] T 7ec9bb1d59ac4b11a1f9b0d44b1dc69b P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.443225 32030 consensus_peers.cc:485] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.443325 31899 connection.cc:535] server connection from 127.22.33.65:55927 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.444169 32294 consensus_peers.cc:485] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444288 32294 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444344 31898 consensus_peers.cc:485] T c8020272a8b14dea8a6a82802ea4e10a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444373 32294 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444449 31898 consensus_peers.cc:485] T 5893a54b753f4d52a900d34140f6363d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444475 32294 consensus_peers.cc:485] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444526 31898 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.444567 32294 consensus_peers.cc:485] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.446116 32030 consensus_peers.cc:485] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.450706 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:44.452636 29416 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:44.459738 32030 consensus_peers.cc:485] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.505455 29570 consensus_peers.cc:485] T 3d8ce8f6f27b4ab19ff5a1bb972ca202 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.620911 32030 consensus_peers.cc:485] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.889187 32030 consensus_peers.cc:485] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.896092 29311 consensus_peers.cc:485] T 0efddc01769f43e0b6f44d40ce6fa562 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:44.896320 29311 consensus_peers.cc:485] T 70ffd2100f104b339bfdc2d272ba3729 P b87d188552494b4590afc37a85b98c35 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:44.919124 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29295
I20240501 14:02:44.927814 14025 raft_consensus.cc:482] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:44.928041 14025 raft_consensus.cc:504] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:44.928818 14025 leader_election.cc:288] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:44.930378 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d3038e25f4f7793ba0f8c9bd64678" candidate_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
W20240501 14:02:44.933099 29570 leader_election.cc:334] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:44.933259 29570 leader_election.cc:302] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecc8bf0965164f199fb0bfa3d7b47c0e; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6a446ac802a7457d9f27076781f21d79
I20240501 14:02:44.933698 15376 raft_consensus.cc:2726] T 380d3038e25f4f7793ba0f8c9bd64678 P ecc8bf0965164f199fb0bfa3d7b47c0e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:44.988437 15496 raft_consensus.cc:482] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
W20240501 14:02:44.988517 32163 connection.cc:535] client connection to 127.22.33.66:39595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.988579 32030 connection.cc:535] server connection from 127.22.33.66:42885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.988744 31900 connection.cc:535] server connection from 127.22.33.66:57827 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:44.988684 15496 raft_consensus.cc:504] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:44.989482 15496 leader_election.cc:288] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:44.989473 32296 connection.cc:535] client connection to 127.22.33.66:39595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.989949 29572 connection.cc:535] client connection to 127.22.33.66:39595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:44.990341 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29555
W20240501 14:02:44.991865 15497 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:44.989207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.33.66:39595 (local address 127.22.33.71:34641)
0501 14:02:44.991036 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:44.991113 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1701}
W20240501 14:02:44.991971 31900 connection.cc:535] client connection to 127.22.33.66:39595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.992128 32163 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.992251 32163 consensus_peers.cc:485] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.992349 32163 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.992460 32163 consensus_peers.cc:485] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.992941 31900 consensus_peers.cc:485] T bfbf43ae81ae45ba8d5abd345d18ac8a P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993098 31900 consensus_peers.cc:485] T b516ab873c134b79a4424f5ffbc471fb P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993183 31900 consensus_peers.cc:485] T 0fdebc6e2fdd4b2daa399cd1301fcde8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993213 29572 consensus_peers.cc:485] T 241a2fdfe954476ba9bd2cfeb4b931ac P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993263 31900 consensus_peers.cc:485] T fa4a1cdf1c214e929b1d81ae3df585d9 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993353 29572 consensus_peers.cc:485] T a82cbc0696fe4a8a815eaf0538b5aad8 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993357 31900 consensus_peers.cc:485] T e6f045ec0d8448dc9fed9f14a7990170 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993476 29572 consensus_peers.cc:485] T e5e2a1f3418c477aa65b77bfa171a5a1 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993575 29572 consensus_peers.cc:485] T 0d2580d876e543549ca1c5677ccbf32c P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993703 29572 consensus_peers.cc:485] T 34d3876037334884a4a36bc192956cf4 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.993824 29572 consensus_peers.cc:485] T 77ce342cbfad4419b3e2e8d3a52c4628 P ecc8bf0965164f199fb0bfa3d7b47c0e -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.994400 32161 leader_election.cc:334] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:44.994612 32296 consensus_peers.cc:485] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.994738 32296 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.994827 32296 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.994920 32296 consensus_peers.cc:485] T e83b7b4decde4e459e49de9eb05b8c0f P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.995007 32296 consensus_peers.cc:485] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:44.994971 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:02:44.995097 32296 consensus_peers.cc:485] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:44.995093 32032 connection.cc:535] client connection to 127.22.33.66:39595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:44.996007 32032 consensus_peers.cc:485] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:44.996400 32162 leader_election.cc:302] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:44.996728 15496 raft_consensus.cc:2726] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.001494 29676 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:45.001917 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:45.041569 31899 consensus_peers.cc:485] T 3f3d8b11a9bd4f69a855aa087e7cbb5d P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:45.041679 15492 raft_consensus.cc:482] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.041859 15492 raft_consensus.cc:504] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.042716 15492 leader_election.cc:288] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:45.043167 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:02:45.043926 32294 leader_election.cc:334] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.044045 32294 leader_election.cc:302] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:45.044268 15492 raft_consensus.cc:2726] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.051041 15256 raft_consensus.cc:482] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.051198 15256 raft_consensus.cc:504] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:02:45.051805 15256 leader_election.cc:288] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:45.053856 32030 leader_election.cc:334] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:45.063483 32032 leader_election.cc:334] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.063612 32032 leader_election.cc:302] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.063855 15256 raft_consensus.cc:2726] T 1a412c988fd946d7b7d51606be01d1fa P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.114815 15488 raft_consensus.cc:482] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 522f4ff5e9084cbf8462196533c7d7aa)
I20240501 14:02:45.115017 15488 raft_consensus.cc:504] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.115785 15488 leader_election.cc:288] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
W20240501 14:02:45.117038 32031 leader_election.cc:341] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615): Not found: Tablet not found: 49b72e2ac9df4426b071a7e5004e12e1
W20240501 14:02:45.117208 32031 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:45.117283 32031 leader_election.cc:302] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:45.117472 15488 raft_consensus.cc:2726] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.119045 15492 raft_consensus.cc:482] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.119179 15492 raft_consensus.cc:504] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.119716 15492 leader_election.cc:288] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:45.121117 32295 leader_election.cc:334] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:45.121327 32294 leader_election.cc:334] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.121421 32294 leader_election.cc:302] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 522f4ff5e9084cbf8462196533c7d7aa, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:45.121663 15492 raft_consensus.cc:2726] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.125182 15492 raft_consensus.cc:482] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.125319 15492 raft_consensus.cc:504] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.126037 15492 leader_election.cc:288] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:45.126534 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:02:45.126744 32222 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:02:45.127105 32295 leader_election.cc:302] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
I20240501 14:02:45.127365 15492 raft_consensus.cc:2781] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:45.127441 15492 raft_consensus.cc:482] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.127512 15492 raft_consensus.cc:3037] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:45.130213 15492 raft_consensus.cc:504] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.131479 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500b9c9bdfa649bd883d6ef2d6e541fc" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:02:45.131664 32222 raft_consensus.cc:3037] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:45.132843 32294 leader_election.cc:334] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:45.133641 32294 leader_election.cc:334] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.143312 15492 leader_election.cc:288] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:45.152135 32222 raft_consensus.cc:2454] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:02:45.152676 32295 leader_election.cc:302] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:45.153024 15492 raft_consensus.cc:2781] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:45.153282 15492 raft_consensus.cc:686] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [term 2 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:02:45.153564 15492 consensus_queue.cc:237] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.158108 29103 catalog_manager.cc:5484] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 1 to 2, leader changed from 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 2 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
I20240501 14:02:45.169345 15492 raft_consensus.cc:482] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.169512 15492 raft_consensus.cc:504] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.170126 15492 leader_election.cc:288] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:45.170740 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec283b44ec4632b620275af509565e" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:02:45.171890 32294 leader_election.cc:334] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.172019 32294 leader_election.cc:302] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:45.172238 15492 raft_consensus.cc:2726] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.200616 15496 raft_consensus.cc:482] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.200807 15496 raft_consensus.cc:504] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:45.201565 15496 leader_election.cc:288] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.202764 32161 leader_election.cc:334] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:45.214460 31899 consensus_peers.cc:485] T 86b4f0fc35a2401a9b80c3d590e6dbb8 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:45.226150 32032 consensus_peers.cc:485] T f345c7a43d2648b4b4c47e726c4d4c51 P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.235496 15486 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.235663 15486 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.236403 15486 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:45.236867 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a697f3a1ee544daa7073814ab845d58" is_pre_election: true
W20240501 14:02:45.237668 31898 leader_election.cc:334] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.237768 31898 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 0a697f3a1ee544daa7073814ab845d58, 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:45.238023 15486 raft_consensus.cc:2726] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:45.240142 32031 consensus_peers.cc:485] T 54bdc21d1dc644e7ac44f5b9ad39ab32 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:02:45.249080 15496 raft_consensus.cc:482] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.249246 15496 raft_consensus.cc:504] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.249958 15496 leader_election.cc:288] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:45.250759 31963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9320d3f885be46e4bf165243eaef41aa" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
W20240501 14:02:45.251590 32161 leader_election.cc:334] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.251786 32163 leader_election.cc:302] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 9903ed35bca144bcb74e9bf5102117e7
I20240501 14:02:45.252115 15496 raft_consensus.cc:2726] T 9320d3f885be46e4bf165243eaef41aa P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.282954 15488 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.283143 15488 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.283963 15488 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:45.284500 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7" is_pre_election: true
I20240501 14:02:45.284736 31965 raft_consensus.cc:2454] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 1.
I20240501 14:02:45.285113 32032 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:02:45.285387 15488 raft_consensus.cc:2781] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:45.285503 15488 raft_consensus.cc:482] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.285607 15488 raft_consensus.cc:3037] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:45.285838 32030 leader_election.cc:334] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.287638 15488 raft_consensus.cc:504] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.288136 15488 leader_election.cc:288] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:45.288625 31965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df490207f6248ce8a36687d71fedb85" candidate_uuid: "0a697f3a1ee544daa7073814ab845d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9903ed35bca144bcb74e9bf5102117e7"
I20240501 14:02:45.288802 31965 raft_consensus.cc:3037] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:45.289896 32030 leader_election.cc:334] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.291514 31965 raft_consensus.cc:2454] T 3df490207f6248ce8a36687d71fedb85 P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697f3a1ee544daa7073814ab845d58 in term 2.
I20240501 14:02:45.291869 32032 leader_election.cc:302] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58, 9903ed35bca144bcb74e9bf5102117e7; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:45.292306 15488 raft_consensus.cc:2781] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:45.292474 15488 raft_consensus.cc:686] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [term 2 LEADER]: Becoming Leader. State: Replica: 0a697f3a1ee544daa7073814ab845d58, State: Running, Role: LEADER
I20240501 14:02:45.292724 15488 consensus_queue.cc:237] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.297475 29103 catalog_manager.cc:5484] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 reported cstate change: term changed from 1 to 2, leader changed from 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) to 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70). New cstate: current_term: 2 leader_uuid: "0a697f3a1ee544daa7073814ab845d58" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } }
I20240501 14:02:45.304396 15492 raft_consensus.cc:482] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.304591 15492 raft_consensus.cc:504] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:45.305333 15492 leader_election.cc:288] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.306443 32294 leader_election.cc:334] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.331003 15486 raft_consensus.cc:482] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.331154 15486 raft_consensus.cc:504] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.331848 15486 leader_election.cc:288] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.333549 31898 leader_election.cc:334] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.400956 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29555
I20240501 14:02:45.411235 15341 raft_consensus.cc:482] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.411453 15341 raft_consensus.cc:504] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.412257 15341 leader_election.cc:288] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:45.412693 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6a446ac802a7457d9f27076781f21d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
W20240501 14:02:45.413640 32161 leader_election.cc:334] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.413743 32161 leader_election.cc:302] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, 6edd6a3430d44c38a19c2804e555059f
I20240501 14:02:45.413986 15341 raft_consensus.cc:2726] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.420859 15492 raft_consensus.cc:482] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.421087 15492 raft_consensus.cc:504] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.421866 15492 leader_election.cc:288] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:45.422376 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79" is_pre_election: true
I20240501 14:02:45.422587 32222 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 1.
I20240501 14:02:45.423570 32295 leader_election.cc:302] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters:
W20240501 14:02:45.423799 32294 leader_election.cc:334] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.423851 15492 raft_consensus.cc:2781] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:45.423928 15492 raft_consensus.cc:482] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.424007 15492 raft_consensus.cc:3037] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:45.425856 15492 raft_consensus.cc:504] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.426389 15492 leader_election.cc:288] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Requested vote from peers 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:45.427188 32222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a025068d40c4b388421c5e6930847c2" candidate_uuid: "6edd6a3430d44c38a19c2804e555059f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a446ac802a7457d9f27076781f21d79"
I20240501 14:02:45.427386 32222 raft_consensus.cc:3037] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:45.428823 32294 leader_election.cc:334] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.429814 32222 raft_consensus.cc:2454] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edd6a3430d44c38a19c2804e555059f in term 2.
I20240501 14:02:45.430320 32295 leader_election.cc:302] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79, 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba
I20240501 14:02:45.430613 15411 raft_consensus.cc:2781] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:45.430776 15411 raft_consensus.cc:686] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [term 2 LEADER]: Becoming Leader. State: Replica: 6edd6a3430d44c38a19c2804e555059f, State: Running, Role: LEADER
I20240501 14:02:45.431037 15411 consensus_queue.cc:237] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.431932 15492 raft_consensus.cc:482] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.433081 15492 raft_consensus.cc:504] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.433615 15492 leader_election.cc:288] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:45.435024 32296 leader_election.cc:334] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.436556 29103 catalog_manager.cc:5484] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f reported cstate change: term changed from 1 to 2, leader changed from 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) to 6edd6a3430d44c38a19c2804e555059f (127.22.33.72). New cstate: current_term: 2 leader_uuid: "6edd6a3430d44c38a19c2804e555059f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } }
W20240501 14:02:45.437309 32294 leader_election.cc:334] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.437418 32294 leader_election.cc:302] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.437631 15492 raft_consensus.cc:2726] T ca0a888dc37d44c69dea73f9ffd556f6 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.455190 15486 raft_consensus.cc:482] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.455416 15486 raft_consensus.cc:504] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.456393 15486 leader_election.cc:288] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:45.456845 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f" is_pre_election: true
I20240501 14:02:45.457110 32361 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 1.
I20240501 14:02:45.457513 31899 leader_election.cc:302] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:02:45.457985 15486 raft_consensus.cc:2781] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:45.458098 15486 raft_consensus.cc:482] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.458215 15486 raft_consensus.cc:3037] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:45.458341 31898 leader_election.cc:334] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:45.459806 32032 consensus_peers.cc:485] T 815ce94772f2454ea755d33dd04a87d2 P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.460368 15486 raft_consensus.cc:504] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.461154 15488 raft_consensus.cc:482] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.461351 15488 raft_consensus.cc:504] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.461753 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578c3f4c5a7c4e2c962b64736195f0bc" candidate_uuid: "9903ed35bca144bcb74e9bf5102117e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6edd6a3430d44c38a19c2804e555059f"
I20240501 14:02:45.461937 32361 raft_consensus.cc:3037] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:45.462054 15488 leader_election.cc:288] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:45.464711 32361 raft_consensus.cc:2454] T 578c3f4c5a7c4e2c962b64736195f0bc P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9903ed35bca144bcb74e9bf5102117e7 in term 2.
I20240501 14:02:45.465130 31899 leader_election.cc:302] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f, 9903ed35bca144bcb74e9bf5102117e7; no voters:
I20240501 14:02:45.465139 15486 leader_election.cc:288] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: Requested vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 6edd6a3430d44c38a19c2804e555059f (127.22.33.72:37615)
I20240501 14:02:45.465415 15486 raft_consensus.cc:2781] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:02:45.465572 32030 leader_election.cc:334] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.465986 15486 raft_consensus.cc:686] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [term 2 LEADER]: Becoming Leader. State: Replica: 9903ed35bca144bcb74e9bf5102117e7, State: Running, Role: LEADER
W20240501 14:02:45.466434 31898 leader_election.cc:334] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.466305 15486 consensus_queue.cc:237] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
W20240501 14:02:45.467197 32161 connection.cc:535] client connection to 127.22.33.68:34947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:45.467196 32294 connection.cc:535] client connection to 127.22.33.68:34947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:45.467839 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 31881
W20240501 14:02:45.468348 32161 leader_election.cc:334] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:45.468470 32161 leader_election.cc:302] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:02:45.468564 31898 connection.cc:535] client connection to 127.22.33.68:34947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:45.468775 32161 consensus_peers.cc:485] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.468889 32161 consensus_peers.cc:485] T 6b83cf8da83047f8aa304e04d58771a8 P 6a446ac802a7457d9f27076781f21d79 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.468994 32161 consensus_peers.cc:485] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.469120 32030 connection.cc:535] client connection to 127.22.33.68:34947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:45.469481 15341 raft_consensus.cc:2726] T 380d3038e25f4f7793ba0f8c9bd64678 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:45.469825 31898 consensus_peers.cc:485] T b5aa71ecef4d419eb36c7777790e6f08 P 9903ed35bca144bcb74e9bf5102117e7 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.469986 31898 leader_election.cc:334] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:45.470119 31898 leader_election.cc:302] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9903ed35bca144bcb74e9bf5102117e7; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:02:45.470723 32294 consensus_peers.cc:485] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.470867 32294 leader_election.cc:334] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:45.470934 32030 leader_election.cc:334] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:45.470970 32294 leader_election.cc:302] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:45.471096 32030 leader_election.cc:302] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
W20240501 14:02:45.471298 32294 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.471359 15513 raft_consensus.cc:2726] T 60527f81f34b45c199bf52bf0df53ee2 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:45.472222 32030 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.472374 32030 leader_election.cc:334] T 49b72e2ac9df4426b071a7e5004e12e1 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:45.472702 32030 consensus_peers.cc:485] T 0ff2fc8d230a45daa4e0c8ba4fec422e P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.472754 15411 raft_consensus.cc:2726] T e5ed5a5797424f0f88c2b3c711468e6b P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.473031 15488 raft_consensus.cc:2726] T 83ca18280bbb49058e5aa80a3832b666 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:45.472841 32030 consensus_peers.cc:485] T eb982ed7bcf74eb981c5199d87f9b4c4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.473307 32030 consensus_peers.cc:485] T 9342fb2bff81431db52967c7bf4d331a P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.473423 32030 consensus_peers.cc:485] T 12f0264161ec46af90b404aa18da2b62 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.473533 32030 consensus_peers.cc:485] T 7fce4b1a15bc48e9b8042869d5c18d16 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.473608 32030 consensus_peers.cc:485] T 9c7cd18adcd844f4be7ddd343cda3df4 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.473709 32030 consensus_peers.cc:485] T 5c0fc97541db4467b5089bbeaf9bfca0 P 0a697f3a1ee544daa7073814ab845d58 -> Peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Couldn't send request to peer ecc8bf0965164f199fb0bfa3d7b47c0e. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.477133 32162 consensus_peers.cc:485] T 5a8d8249c87f42faa9a7464369048090 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:45.477641 29103 catalog_manager.cc:5484] T 578c3f4c5a7c4e2c962b64736195f0bc P 9903ed35bca144bcb74e9bf5102117e7 reported cstate change: term changed from 1 to 2, leader changed from 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65) to 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69). New cstate: current_term: 2 leader_uuid: "9903ed35bca144bcb74e9bf5102117e7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } }
I20240501 14:02:45.482048 32004 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:45.491206 32032 consensus_peers.cc:485] T 62c84a393bd047d985927417f82bec7d P 0a697f3a1ee544daa7073814ab845d58 -> Peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Couldn't send request to peer b87d188552494b4590afc37a85b98c35. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.521812 15488 raft_consensus.cc:482] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.522013 15488 raft_consensus.cc:504] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.522703 15488 leader_election.cc:288] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.523960 32030 leader_election.cc:334] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:45.524123 32032 leader_election.cc:334] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.524262 32032 leader_election.cc:302] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:45.524525 15488 raft_consensus.cc:2726] T b0d8239317634874b1ffb1ba239bb9ad P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.532783 15488 raft_consensus.cc:482] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.532994 15488 raft_consensus.cc:504] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.533661 15488 leader_election.cc:288] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:45.535037 32031 leader_election.cc:334] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:45.535187 32032 leader_election.cc:334] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.535328 32032 leader_election.cc:302] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.535642 15488 raft_consensus.cc:2726] T 0efddc01769f43e0b6f44d40ce6fa562 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:45.557129 32294 consensus_peers.cc:485] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.569020 32031 consensus_peers.cc:462] T 413a79f6a61a49d9a49a5a52cec4c00f P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:45.646188 32222 raft_consensus.cc:1261] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:02:45.646713 15492 consensus_queue.cc:1035] T 500b9c9bdfa649bd883d6ef2d6e541fc P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:02:45.713883 32162 consensus_peers.cc:485] T ad7af304700545b2a28249f9f046cab6 P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:02:45.746194 32031 consensus_peers.cc:485] T 21c02994dfef4a10a87b1f47d3ab6e3b P 0a697f3a1ee544daa7073814ab845d58 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:02:45.746945 32030 consensus_peers.cc:485] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:45.788899 15488 raft_consensus.cc:482] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.789142 15488 raft_consensus.cc:504] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } }
I20240501 14:02:45.789820 15488 leader_election.cc:288] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.791124 32030 leader_election.cc:334] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:45.791268 32032 leader_election.cc:334] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.791368 32032 leader_election.cc:302] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:45.791642 15488 raft_consensus.cc:2726] T 0e33e4f9b1204797aeff0bca36c93911 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.832648 15486 raft_consensus.cc:482] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.832854 15486 raft_consensus.cc:504] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:45.833544 15486 leader_election.cc:288] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:45.834581 32294 consensus_peers.cc:485] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.841917 31900 leader_election.cc:334] T db862a636e5a427085a2fa31f6fe8050 P 9903ed35bca144bcb74e9bf5102117e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:45.861658 32162 consensus_peers.cc:485] T 8427dee24ad04247b23b24570c73c92a P 6a446ac802a7457d9f27076781f21d79 -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:02:45.870082 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 31881
I20240501 14:02:45.872781 15256 raft_consensus.cc:482] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:45.873028 15256 raft_consensus.cc:504] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:45.874006 15256 leader_election.cc:288] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
W20240501 14:02:45.875720 32030 leader_election.cc:334] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
W20240501 14:02:45.875963 32032 leader_election.cc:334] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.876096 32032 leader_election.cc:302] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.876371 15256 raft_consensus.cc:2726] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.898550 15492 raft_consensus.cc:482] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.898766 15492 raft_consensus.cc:504] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:45.899488 15492 leader_election.cc:288] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:45.901038 32296 leader_election.cc:334] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:45.901196 32295 leader_election.cc:334] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:45.901293 32295 leader_election.cc:302] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.901551 15492 raft_consensus.cc:2726] T 67ed6fab7d9e46aa8faeb3cf784af0fb P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.915186 15256 raft_consensus.cc:482] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.915386 15256 raft_consensus.cc:504] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:45.916060 15256 leader_election.cc:288] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
W20240501 14:02:45.917652 32032 leader_election.cc:334] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:45.917929 32030 leader_election.cc:334] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:45.918064 32030 leader_election.cc:302] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a697f3a1ee544daa7073814ab845d58; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:45.918819 15256 raft_consensus.cc:2726] T 7da6bff08fa14cf6b4894f58e248fb59 P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.928304 15529 raft_consensus.cc:482] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:45.928594 15529 raft_consensus.cc:504] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:45.930526 15529 leader_election.cc:288] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:45.931188 32162 leader_election.cc:334] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:45.932041 15492 raft_consensus.cc:482] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:45.932200 15492 raft_consensus.cc:504] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
W20240501 14:02:45.933403 32163 leader_election.cc:334] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:45.933548 32163 leader_election.cc:302] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35
W20240501 14:02:45.933884 32032 connection.cc:535] client connection to 127.22.33.69:32783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:45.934056 32163 connection.cc:535] server connection from 127.22.33.69:52917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:45.934173 15341 raft_consensus.cc:2726] T 70ffd2100f104b339bfdc2d272ba3729 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:02:45.934438 32296 connection.cc:535] client connection to 127.22.33.69:32783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:45.934592 15492 leader_election.cc:288] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:45.934706 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 32014
W20240501 14:02:45.935463 32032 leader_election.cc:334] T c6973301bdd94c2b8f98e718715ff752 P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111)
W20240501 14:02:45.935593 32296 consensus_peers.cc:485] T 9c859b6e5b674245965fac9c113664b9 P 6edd6a3430d44c38a19c2804e555059f -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.935726 32296 consensus_peers.cc:485] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.935694 32295 leader_election.cc:334] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
W20240501 14:02:45.935760 32032 consensus_peers.cc:485] T 1eb574dc421745618ba2a11f0b3f9110 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.935870 32032 consensus_peers.cc:485] T 3df490207f6248ce8a36687d71fedb85 P 0a697f3a1ee544daa7073814ab845d58 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:45.935843 32294 leader_election.cc:334] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:45.935964 32294 leader_election.cc:302] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:45.936216 15492 raft_consensus.cc:2726] T 9e32147255254f2d8baff46c9b2d0e60 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:45.945010 32137 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:45.973462 32222 raft_consensus.cc:1261] T 1a025068d40c4b388421c5e6930847c2 P 6a446ac802a7457d9f27076781f21d79 [term 2 FOLLOWER]: Refusing update from remote peer 6edd6a3430d44c38a19c2804e555059f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:02:45.974131 15492 consensus_queue.cc:1035] T 1a025068d40c4b388421c5e6930847c2 P 6edd6a3430d44c38a19c2804e555059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:45.992139 32296 consensus_peers.cc:485] T 9c7c0ef199394de490b790a26532df3b P 6edd6a3430d44c38a19c2804e555059f -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.010515 32163 consensus_peers.cc:485] T 25a6e3129a2f4bf89a36cdca5eaeb376 P 6a446ac802a7457d9f27076781f21d79 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:46.162557 15492 raft_consensus.cc:482] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.162784 15492 raft_consensus.cc:504] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
W20240501 14:02:46.165297 32296 leader_election.cc:334] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:46.165839 15492 leader_election.cc:288] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:46.166875 32294 leader_election.cc:334] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:46.167004 32294 leader_election.cc:302] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.167277 15492 raft_consensus.cc:2726] T 241a2fdfe954476ba9bd2cfeb4b931ac P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:46.174822 15529 raft_consensus.cc:482] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.175006 15529 raft_consensus.cc:504] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:02:46.175659 15529 leader_election.cc:288] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:46.177189 32161 leader_election.cc:334] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:46.177613 32163 leader_election.cc:334] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:46.177778 32163 leader_election.cc:302] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.178025 15529 raft_consensus.cc:2726] T 34d3876037334884a4a36bc192956cf4 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.193878 32295 consensus_peers.cc:485] T b12b60d5bf304c28ae2041629b8d5bdd P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:02:46.194308 32163 consensus_peers.cc:485] T 6a9d23c77f4e432980759f1b4c2bebfa P 6a446ac802a7457d9f27076781f21d79 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.231619 32163 consensus_peers.cc:485] T 820be6be18af4781ace626f239379cff P 6a446ac802a7457d9f27076781f21d79 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:46.287478 15256 raft_consensus.cc:482] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.287670 15256 raft_consensus.cc:504] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:46.288364 15256 leader_election.cc:288] T 0d2580d876e543549ca1c5677ccbf32c P 0a697f3a1ee544daa7073814ab845d58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:46.288842 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 32014
W20240501 14:02:46.291375 32163 consensus_peers.cc:485] T c696bdff78504c0c93f39a7e07a40dcd P 6a446ac802a7457d9f27076781f21d79 -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:46.311281 15529 raft_consensus.cc:482] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.311487 15529 raft_consensus.cc:504] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:46.312130 15529 leader_election.cc:288] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:46.313570 32161 leader_election.cc:334] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:46.313798 32162 leader_election.cc:334] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:46.313902 32162 leader_election.cc:302] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.314173 15529 raft_consensus.cc:2726] T 5351f83631e64b7abe60575a6c285592 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.317924 32296 consensus_peers.cc:485] T d2cf9b7f68034fcb80cf620687ca4a81 P 6edd6a3430d44c38a19c2804e555059f -> Peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Couldn't send request to peer 9903ed35bca144bcb74e9bf5102117e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:46.319667 15529 raft_consensus.cc:482] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.319847 15529 raft_consensus.cc:504] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:46.320679 15529 leader_election.cc:288] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
W20240501 14:02:46.321887 32161 leader_election.cc:334] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:46.322039 32163 leader_election.cc:334] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111)
I20240501 14:02:46.322141 32163 leader_election.cc:302] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 9903ed35bca144bcb74e9bf5102117e7, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.322405 15529 raft_consensus.cc:2726] T 192a6cc308da4a448f03717b6a439094 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.337594 32294 consensus_peers.cc:485] T 356722d5ec9d4d3d8ba08b4e7c2b2b11 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:46.348317 32163 connection.cc:535] client connection to 127.22.33.70:37745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:46.348887 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 32146
W20240501 14:02:46.349543 32296 consensus_peers.cc:485] T 66a151a1d8f74a5897dd6c8e4b2a7057 P 6edd6a3430d44c38a19c2804e555059f -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.349680 32296 consensus_peers.cc:485] T 46d49324c56d499a96d7cca9af8a4867 P 6edd6a3430d44c38a19c2804e555059f -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.349781 32296 consensus_peers.cc:485] T 36b0222ebb6d4246ab8551d5579b1760 P 6edd6a3430d44c38a19c2804e555059f -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.349817 32163 consensus_peers.cc:485] T c05d6d2f1f1140c7bea1e084574b51fe P 6a446ac802a7457d9f27076781f21d79 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:46.355068 32269 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:46.408315 15518 raft_consensus.cc:482] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.408432 15492 raft_consensus.cc:482] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.408541 15518 raft_consensus.cc:504] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:46.408617 15492 raft_consensus.cc:504] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:46.409281 15518 leader_election.cc:288] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:46.409296 15492 leader_election.cc:288] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:46.409847 15492 raft_consensus.cc:482] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:46.409987 15492 raft_consensus.cc:504] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:46.410516 15492 leader_election.cc:288] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745)
I20240501 14:02:46.410553 15518 raft_consensus.cc:482] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
W20240501 14:02:46.410686 32295 leader_election.cc:334] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:46.410694 15518 raft_consensus.cc:504] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:46.411295 15518 leader_election.cc:288] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:46.411602 15518 raft_consensus.cc:482] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:46.411784 15518 raft_consensus.cc:504] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } }
I20240501 14:02:46.412382 15518 leader_election.cc:288] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201)
I20240501 14:02:46.412808 15518 raft_consensus.cc:482] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
W20240501 14:02:46.412830 32296 leader_election.cc:334] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.413172 32296 leader_election.cc:334] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:46.413197 15518 raft_consensus.cc:504] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
W20240501 14:02:46.413380 32296 leader_election.cc:334] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111)
W20240501 14:02:46.413569 32294 leader_election.cc:334] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:46.413569 32296 leader_election.cc:302] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 0a697f3a1ee544daa7073814ab845d58, b87d188552494b4590afc37a85b98c35
W20240501 14:02:46.413738 32294 leader_election.cc:334] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:46.413836 32294 leader_election.cc:302] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 522f4ff5e9084cbf8462196533c7d7aa, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.413870 15492 raft_consensus.cc:2726] T 71933eccec5f4fdc8b1df7111f941777 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:46.414039 15492 raft_consensus.cc:2726] T d2e5851626b54e1caf6105a70ccafc43 P 6edd6a3430d44c38a19c2804e555059f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:02:46.414532 32296 leader_election.cc:334] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
I20240501 14:02:46.414566 15518 leader_election.cc:288] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
W20240501 14:02:46.415310 32294 leader_election.cc:334] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:46.415423 32294 leader_election.cc:302] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.416244 15542 raft_consensus.cc:2726] T 798c9c25344d4beb9a079fb776044a17 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.416487 32296 leader_election.cc:334] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.416570 32294 leader_election.cc:334] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:46.416689 32294 leader_election.cc:302] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:46.416906 15542 raft_consensus.cc:2726] T c2c63489706e4d068adde1872f451ad2 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.416968 32294 leader_election.cc:334] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111)
I20240501 14:02:46.417116 32294 leader_election.cc:302] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: 6518b7a04854458bb2fb23edc8b6f7ba, b87d188552494b4590afc37a85b98c35
I20240501 14:02:46.417340 15542 raft_consensus.cc:2726] T eb3c16421adb477e821495e785849cc4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.417882 32294 leader_election.cc:334] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:46.417997 32294 leader_election.cc:302] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.418247 15542 raft_consensus.cc:2726] T 68f3644410a24fc1a9457ff46a10c180 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:46.463867 15542 raft_consensus.cc:482] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.464108 15542 raft_consensus.cc:504] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } }
I20240501 14:02:46.464789 15542 leader_election.cc:288] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
W20240501 14:02:46.466450 32296 leader_election.cc:334] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.466584 32294 leader_election.cc:334] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
I20240501 14:02:46.466719 32294 leader_election.cc:302] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6edd6a3430d44c38a19c2804e555059f; no voters: b87d188552494b4590afc37a85b98c35, ecc8bf0965164f199fb0bfa3d7b47c0e
I20240501 14:02:46.466962 15542 raft_consensus.cc:2726] T e5e2a1f3418c477aa65b77bfa171a5a1 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.529299 32294 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:46.536859 32295 consensus_peers.cc:485] T 6fbb60619c1a4d8e8cd6266596a0614d P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:02:46.605410 32294 consensus_peers.cc:485] T 67623891d415411a9da7d7ae9523f69a P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:02:46.624399 32294 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201): Couldn't send request to peer 6518b7a04854458bb2fb23edc8b6f7ba. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.65:43201: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:02:46.678594 15529 raft_consensus.cc:482] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
W20240501 14:02:46.681682 32163 consensus_peers.cc:485] T c23d0b871aff4ce3850e45ae31097c03 P 6a446ac802a7457d9f27076781f21d79 -> Peer 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745): Couldn't send request to peer 0a697f3a1ee544daa7073814ab845d58. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.70:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:46.681867 32163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.70:37745, user_credentials={real_user=slave}} blocked reactor thread for 326625us
I20240501 14:02:46.683679 15341 raft_consensus.cc:482] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:46.683677 15529 raft_consensus.cc:504] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:46.683921 15341 raft_consensus.cc:504] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:46.684477 15341 leader_election.cc:288] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:46.684600 15529 leader_election.cc:288] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595)
I20240501 14:02:46.684610 15341 raft_consensus.cc:482] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b87d188552494b4590afc37a85b98c35)
I20240501 14:02:46.684791 15341 raft_consensus.cc:504] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:46.685338 15341 leader_election.cc:288] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:46.684794 15529 raft_consensus.cc:482] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9903ed35bca144bcb74e9bf5102117e7)
I20240501 14:02:46.685921 15529 raft_consensus.cc:504] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "b87d188552494b4590afc37a85b98c35" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 39595 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:46.686786 15529 leader_election.cc:288] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:46.690759 32163 leader_election.cc:334] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.690935 32163 leader_election.cc:334] T a6e94a79c4514fe4a3ae8e41c7792dc9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.691068 32163 leader_election.cc:334] T fe99c587804f47c3b58ec3ddbab8a371 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.693289 32163 leader_election.cc:334] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b87d188552494b4590afc37a85b98c35 (127.22.33.66:39595): Network error: Client connection negotiation failed: client connection to 127.22.33.66:39595: connect: Connection refused (error 111)
W20240501 14:02:46.693603 32163 leader_election.cc:334] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783): Network error: Client connection negotiation failed: client connection to 127.22.33.69:32783: connect: Connection refused (error 111)
I20240501 14:02:46.693792 32163 leader_election.cc:302] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 9903ed35bca144bcb74e9bf5102117e7, b87d188552494b4590afc37a85b98c35
I20240501 14:02:46.694132 15529 raft_consensus.cc:2726] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.695035 32162 leader_election.cc:334] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:46.695178 32162 leader_election.cc:302] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a446ac802a7457d9f27076781f21d79; no voters: 522f4ff5e9084cbf8462196533c7d7aa, b87d188552494b4590afc37a85b98c35
I20240501 14:02:46.695478 15529 raft_consensus.cc:2726] T a87b1de51c4c4f909c27c27aa745fb44 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:46.696441 32162 leader_election.cc:334] T fa4a1cdf1c214e929b1d81ae3df585d9 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111)
I20240501 14:02:46.697364 15341 raft_consensus.cc:482] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.697521 15341 raft_consensus.cc:504] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:46.698311 15341 leader_election.cc:288] T 8a1690e3a76644859a0711e305b8f946 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:46.699872 15341 raft_consensus.cc:482] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9903ed35bca144bcb74e9bf5102117e7)
I20240501 14:02:46.700044 15341 raft_consensus.cc:504] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } }
I20240501 14:02:46.700695 15341 leader_election.cc:288] T ba3bd81448244b5bbdcd9eb8d23aee03 P 6a446ac802a7457d9f27076781f21d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
I20240501 14:02:46.701856 15341 raft_consensus.cc:482] T a82cbc0696fe4a8a815eaf0538b5aad8 P 6a446ac802a7457d9f27076781f21d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecc8bf0965164f199fb0bfa3d7b47c0e)
I20240501 14:02:46.702003 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 32146
W20240501 14:02:46.736070 32295 consensus_peers.cc:485] T b36d195f6135436293834aec4f3296a2 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:02:46.755652 32295 consensus_peers.cc:485] T 3ec202bb41a44e05939f1ecdf33d8c38 P 6edd6a3430d44c38a19c2804e555059f -> Peer 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187): Couldn't send request to peer 522f4ff5e9084cbf8462196533c7d7aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.33.67:44187: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:02:46.756428 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 32277
I20240501 14:02:46.762766 32402 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:47.085505 15492 raft_consensus.cc:482] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9903ed35bca144bcb74e9bf5102117e7)
I20240501 14:02:47.085805 15492 raft_consensus.cc:504] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } }
I20240501 14:02:47.086709 15492 leader_election.cc:288] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:47.110014 15492 raft_consensus.cc:482] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a697f3a1ee544daa7073814ab845d58)
I20240501 14:02:47.117028 15518 raft_consensus.cc:482] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a697f3a1ee544daa7073814ab845d58)
I20240501 14:02:47.117277 15518 raft_consensus.cc:504] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "ecc8bf0965164f199fb0bfa3d7b47c0e" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 34947 } }
I20240501 14:02:47.118095 15518 leader_election.cc:288] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947)
I20240501 14:02:47.118283 15546 raft_consensus.cc:482] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:47.118305 15492 raft_consensus.cc:504] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697f3a1ee544daa7073814ab845d58" member_type: VOTER last_known_addr { host: "127.22.33.70" port: 37745 } } peers { permanent_uuid: "6a446ac802a7457d9f27076781f21d79" member_type: VOTER last_known_addr { host: "127.22.33.71" port: 38743 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:47.118433 15546 raft_consensus.cc:504] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } } peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "9903ed35bca144bcb74e9bf5102117e7" member_type: VOTER last_known_addr { host: "127.22.33.69" port: 32783 } }
I20240501 14:02:47.118870 15492 leader_election.cc:288] T 054b34b2b424475ca4310b6dd9d97e8e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a697f3a1ee544daa7073814ab845d58 (127.22.33.70:37745), 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743)
I20240501 14:02:47.118952 15546 leader_election.cc:288] T 83ec283b44ec4632b620275af509565e P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 9903ed35bca144bcb74e9bf5102117e7 (127.22.33.69:32783)
I20240501 14:02:47.119027 15551 raft_consensus.cc:482] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6518b7a04854458bb2fb23edc8b6f7ba)
I20240501 14:02:47.119164 15551 raft_consensus.cc:504] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6518b7a04854458bb2fb23edc8b6f7ba" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43201 } } peers { permanent_uuid: "522f4ff5e9084cbf8462196533c7d7aa" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44187 } } peers { permanent_uuid: "6edd6a3430d44c38a19c2804e555059f" member_type: VOTER last_known_addr { host: "127.22.33.72" port: 37615 } }
I20240501 14:02:47.119668 15551 leader_election.cc:288] T 66f7d82d97b6414992f7fdcdaeb83e3d P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6518b7a04854458bb2fb23edc8b6f7ba (127.22.33.65:43201), 522f4ff5e9084cbf8462196533c7d7aa (127.22.33.67:44187)
W20240501 14:02:47.121835 32295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44187, user_credentials={real_user=slave}} blocked reactor thread for 366204us
W20240501 14:02:47.122087 32294 leader_election.cc:334] T 9c7cd18adcd844f4be7ddd343cda3df4 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecc8bf0965164f199fb0bfa3d7b47c0e (127.22.33.68:34947): Network error: Client connection negotiation failed: client connection to 127.22.33.68:34947: connect: Connection refused (error 111)
W20240501 14:02:47.122813 32295 connection.cc:698] client connection to 127.22.33.71:38743 send error: Network error: sendmsg error: Broken pipe (error 32)
W20240501 14:02:47.124789 32295 leader_election.cc:334] T 59c54188ad9e40dd925b50a250dcea04 P 6edd6a3430d44c38a19c2804e555059f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6a446ac802a7457d9f27076781f21d79 (127.22.33.71:38743): Network error: Client connection negotiation failed: client connection to 127.22.33.71:38743: connect: Connection refused (error 111)
I20240501 14:02:47.125051 15551 raft_consensus.cc:482] T 7fce4b1a15bc48e9b8042869d5c18d16 P 6edd6a3430d44c38a19c2804e555059f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a697f3a1ee544daa7073814ab845d58)
I20240501 14:02:47.125105 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 32277
I20240501 14:02:47.174901 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 29074
I20240501 14:02:47.181272 29135 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:47.475957 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 29074
2024-05-01T14:02:47Z chronyd exiting
I20240501 14:02:47.545902 22661 test_util.cc:175] -----------------------------------------------
I20240501 14:02:47.545992 22661 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714571948533166-22661-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (163560 ms)
[----------] 1 test from DDLDuringRebalancingTest (163560 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240501 14:02:47.548153 22661 test_util.cc:268] Using random seed: -251851564
I20240501 14:02:47.549959 22661 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:02:47.550030 22661 ts_itest-base.cc:113] --------------
I20240501 14:02:47.550087 22661 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:02:47.550153 22661 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:02:47.550201 22661 ts_itest-base.cc:116] --------------
2024-05-01T14:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:47Z Disabled control of system clock
I20240501 14:02:47.582353 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:42697
--webserver_interface=127.22.33.126
--webserver_port=0
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.33.126:42697
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:02:47.934096 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:47.934443 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:47.934578 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:47.942787 15564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:47.942958 15564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:47.943078 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:47.943161 15564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:47.943238 15564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:47.953333 15564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.33.126:42697
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.33.126:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.33.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:47.954759 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:47.956665 15564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:47.968664 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:47.969205 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:47.969578 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:47.970427 15564 server_base.cc:1034] running on GCE node
I20240501 14:02:47.971324 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:47.973227 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:47.974479 15564 hybrid_clock.cc:648] HybridClock initialized: now 1714572167974433 us; error 45 us; skew 500 ppm
I20240501 14:02:47.974833 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:47.977274 15564 webserver.cc:460] Webserver started at http://127.22.33.126:42135/ using document root <none> and password file <none>
I20240501 14:02:47.978025 15564 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:47.978127 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:47.978413 15564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:47.981132 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "19cbe1cda82142e5bcac503c126e98d7"
format_stamp: "Formatted at 2024-05-01 14:02:47 on dist-test-slave-3610"
I20240501 14:02:47.981892 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "19cbe1cda82142e5bcac503c126e98d7"
format_stamp: "Formatted at 2024-05-01 14:02:47 on dist-test-slave-3610"
I20240501 14:02:47.987306 15564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:47.990737 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:47.992101 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:47.992280 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
uuid: "19cbe1cda82142e5bcac503c126e98d7"
format_stamp: "Formatted at 2024-05-01 14:02:47 on dist-test-slave-3610"
I20240501 14:02:47.992483 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:48.023228 15564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:48.024106 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:48.024380 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:48.042598 15564 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.126:42697
I20240501 14:02:48.042603 15629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.126:42697 every 8 connection(s)
I20240501 14:02:48.044041 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:02:48.044351 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 15564
I20240501 14:02:48.044546 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:02:48.048720 15630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:48.061614 15630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7: Bootstrap starting.
I20240501 14:02:48.064639 15630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:48.066016 15630 log.cc:824] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:48.068504 15630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7: No bootstrap required, opened a new log
I20240501 14:02:48.075042 15630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } }
I20240501 14:02:48.075397 15630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:48.075477 15630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cbe1cda82142e5bcac503c126e98d7, State: Initialized, Role: FOLLOWER
I20240501 14:02:48.076032 15630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } }
I20240501 14:02:48.076205 15630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:48.076297 15630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:48.076411 15630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:48.078481 15630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } }
I20240501 14:02:48.078979 15630 leader_election.cc:302] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19cbe1cda82142e5bcac503c126e98d7; no voters:
I20240501 14:02:48.079514 15630 leader_election.cc:288] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:48.079674 15635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:48.080327 15635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [term 1 LEADER]: Becoming Leader. State: Replica: 19cbe1cda82142e5bcac503c126e98d7, State: Running, Role: LEADER
I20240501 14:02:48.080879 15635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } }
I20240501 14:02:48.081684 15630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:48.083861 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19cbe1cda82142e5bcac503c126e98d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } } }
I20240501 14:02:48.084477 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:48.083922 15637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19cbe1cda82142e5bcac503c126e98d7. Latest consensus state: current_term: 1 leader_uuid: "19cbe1cda82142e5bcac503c126e98d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cbe1cda82142e5bcac503c126e98d7" member_type: VOTER last_known_addr { host: "127.22.33.126" port: 42697 } } }
I20240501 14:02:48.089691 15637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:48.090572 15644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:48.096083 15644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:48.105546 15644 catalog_manager.cc:1347] Generated new cluster ID: f7ae4de756b944aa8cdbdb5442ae9f48
I20240501 14:02:48.105672 15644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:48.152863 15644 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:48.154058 15644 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:48.160931 15644 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 19cbe1cda82142e5bcac503c126e98d7: Generated new TSK 0
I20240501 14:02:48.161851 15644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:48.169027 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.65:0
--local_ip_for_outbound_sockets=127.22.33.65
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:02:48.516755 15654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:48.517213 15654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:48.517385 15654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:48.529366 15654 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:48.529536 15654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:48.529750 15654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.65
I20240501 14:02:48.543995 15654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.33.65
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.65
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:48.545790 15654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:48.547878 15654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:48.561050 15661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:48.561455 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:48.561939 15663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:48.562606 15654 server_base.cc:1034] running on GCE node
I20240501 14:02:48.563292 15654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:48.564296 15654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:48.565503 15654 hybrid_clock.cc:648] HybridClock initialized: now 1714572168565450 us; error 42 us; skew 500 ppm
I20240501 14:02:48.565846 15654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:48.568217 15654 webserver.cc:460] Webserver started at http://127.22.33.65:34425/ using document root <none> and password file <none>
I20240501 14:02:48.569033 15654 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:48.569167 15654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:48.569505 15654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:48.572237 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "28461d3f709145e29fc3d3d1c7cce205"
format_stamp: "Formatted at 2024-05-01 14:02:48 on dist-test-slave-3610"
I20240501 14:02:48.573115 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "28461d3f709145e29fc3d3d1c7cce205"
format_stamp: "Formatted at 2024-05-01 14:02:48 on dist-test-slave-3610"
I20240501 14:02:48.578478 15654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 14:02:48.582132 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:48.583725 15654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:48.583925 15654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "28461d3f709145e29fc3d3d1c7cce205"
format_stamp: "Formatted at 2024-05-01 14:02:48 on dist-test-slave-3610"
I20240501 14:02:48.584131 15654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:48.605818 15654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:48.606741 15654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:48.607023 15654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:48.607990 15654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:48.609560 15654 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:48.609658 15654 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:48.609772 15654 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:48.609853 15654 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:48.640791 15654 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.65:43261
I20240501 14:02:48.640858 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.65:43261 every 8 connection(s)
I20240501 14:02:48.642190 15654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:02:48.644502 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 15654
I20240501 14:02:48.644702 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:02:48.648269 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.66:0
--local_ip_for_outbound_sockets=127.22.33.66
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:48.656255 15782 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:48.656612 15782 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:48.657644 15782 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:48.660135 15595 ts_manager.cc:194] Registered new tserver with Master: 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:48.662442 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.65:51107
W20240501 14:02:48.986371 15786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:48.986660 15786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:48.986763 15786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:48.994359 15786 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:48.994482 15786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:48.994604 15786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.66
I20240501 14:02:49.003995 15786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.33.66
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.66
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:49.005440 15786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:49.007207 15786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:49.019372 15792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.019835 15793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.020538 15795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:49.021384 15786 server_base.cc:1034] running on GCE node
I20240501 14:02:49.021956 15786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:49.022847 15786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:49.024034 15786 hybrid_clock.cc:648] HybridClock initialized: now 1714572169023976 us; error 51 us; skew 500 ppm
I20240501 14:02:49.024327 15786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:49.026566 15786 webserver.cc:460] Webserver started at http://127.22.33.66:40083/ using document root <none> and password file <none>
I20240501 14:02:49.027287 15786 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:49.027405 15786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:49.027732 15786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:49.030380 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.031184 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.036275 15786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20240501 14:02:49.039773 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.041208 15786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:49.041354 15786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.041543 15786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:49.075522 15786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:49.076435 15786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:49.076699 15786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:49.077667 15786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:49.079138 15786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:49.079227 15786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.079344 15786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:49.079418 15786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.110548 15786 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.66:40371
I20240501 14:02:49.110616 15913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.66:40371 every 8 connection(s)
I20240501 14:02:49.111863 15786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:49.115089 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 15786
I20240501 14:02:49.115273 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:02:49.118404 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.67:0
--local_ip_for_outbound_sockets=127.22.33.67
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:49.124931 15914 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:49.125324 15914 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:49.126228 15914 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:49.127550 15595 ts_manager.cc:194] Registered new tserver with Master: 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:49.128276 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.66:47863
W20240501 14:02:49.445701 15917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:49.446050 15917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:49.446179 15917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:49.454008 15917 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:49.454170 15917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:49.454351 15917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.67
I20240501 14:02:49.463600 15917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.33.67
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.67
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:49.465013 15917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:49.466768 15917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:49.478941 15926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.479490 15922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.480229 15923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:49.480813 15917 server_base.cc:1034] running on GCE node
I20240501 14:02:49.481436 15917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:49.482412 15917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:49.483613 15917 hybrid_clock.cc:648] HybridClock initialized: now 1714572169483567 us; error 40 us; skew 500 ppm
I20240501 14:02:49.483922 15917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:49.486158 15917 webserver.cc:460] Webserver started at http://127.22.33.67:34711/ using document root <none> and password file <none>
I20240501 14:02:49.486900 15917 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:49.487020 15917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:49.487341 15917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:49.490016 15917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "09ee1930dbed4ac897d146bf885d0e87"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.490749 15917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "09ee1930dbed4ac897d146bf885d0e87"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.495975 15917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:49.499413 15932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.500892 15917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:49.501096 15917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "09ee1930dbed4ac897d146bf885d0e87"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.501294 15917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:49.517823 15917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:49.518647 15917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:49.518913 15917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:49.519870 15917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:49.521399 15917 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:49.521497 15917 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.521608 15917 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:49.521680 15917 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.551640 15917 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.67:44773
I20240501 14:02:49.551685 16044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.67:44773 every 8 connection(s)
I20240501 14:02:49.552932 15917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:49.562052 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 15917
I20240501 14:02:49.562263 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:02:49.565209 16045 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:49.565544 16045 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:49.565652 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.68:0
--local_ip_for_outbound_sockets=127.22.33.68
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:49.566480 16045 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:49.568048 15595 ts_manager.cc:194] Registered new tserver with Master: 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:02:49.568836 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.67:41637
I20240501 14:02:49.666235 15782 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
W20240501 14:02:49.901683 16048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:49.901994 16048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:49.902141 16048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:49.909770 16048 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:49.909914 16048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:49.910094 16048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.68
I20240501 14:02:49.919335 16048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.33.68
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.68
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:49.920692 16048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:49.922971 16048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:49.935523 16053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.936013 16054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:49.936462 16056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:49.937074 16048 server_base.cc:1034] running on GCE node
I20240501 14:02:49.937727 16048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:49.938691 16048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:49.939955 16048 hybrid_clock.cc:648] HybridClock initialized: now 1714572169939897 us; error 44 us; skew 500 ppm
I20240501 14:02:49.940333 16048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:49.942826 16048 webserver.cc:460] Webserver started at http://127.22.33.68:38637/ using document root <none> and password file <none>
I20240501 14:02:49.943614 16048 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:49.943760 16048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:49.944100 16048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:49.947049 16048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.947889 16048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.953519 16048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:02:49.962082 16062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:49.963619 16048 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20240501 14:02:49.963774 16048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
format_stamp: "Formatted at 2024-05-01 14:02:49 on dist-test-slave-3610"
I20240501 14:02:49.963979 16048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:49.998211 16048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:49.999100 16048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:49.999351 16048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:50.000293 16048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:50.001909 16048 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:50.001997 16048 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.002082 16048 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:50.002136 16048 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.034065 16048 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.68:42895
I20240501 14:02:50.034138 16174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.68:42895 every 8 connection(s)
I20240501 14:02:50.035475 16048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:50.038766 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 16048
I20240501 14:02:50.038970 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:02:50.042472 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:0
--local_ip_for_outbound_sockets=127.22.33.69
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:50.052346 16175 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:50.052716 16175 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.053751 16175 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:50.055338 15595 ts_manager.cc:194] Registered new tserver with Master: d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:02:50.056519 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.68:38389
I20240501 14:02:50.130679 15914 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
W20240501 14:02:50.385426 16178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:50.385738 16178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:50.385866 16178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:50.393411 16178 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:50.393571 16178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:50.393702 16178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 14:02:50.402750 16178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:50.404104 16178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:50.405853 16178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:50.417814 16184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.418292 16186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.418763 16183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:50.419371 16178 server_base.cc:1034] running on GCE node
I20240501 14:02:50.419956 16178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:50.420864 16178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:50.422116 16178 hybrid_clock.cc:648] HybridClock initialized: now 1714572170422067 us; error 41 us; skew 500 ppm
I20240501 14:02:50.422497 16178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:50.425418 16178 webserver.cc:460] Webserver started at http://127.22.33.69:44523/ using document root <none> and password file <none>
I20240501 14:02:50.426370 16178 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:50.426503 16178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:50.426890 16178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:50.430202 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "acf31b7bd4974dc4951935d1c2da8f26"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.430985 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "acf31b7bd4974dc4951935d1c2da8f26"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.436676 16178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:02:50.440063 16192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.441432 16178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:50.441591 16178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "acf31b7bd4974dc4951935d1c2da8f26"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.441792 16178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:50.459245 16178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:50.460147 16178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:50.460413 16178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:50.461436 16178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:50.462932 16178 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:50.463028 16178 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.463151 16178 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:50.463224 16178 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.495152 16178 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:37923
I20240501 14:02:50.495200 16304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:37923 every 8 connection(s)
I20240501 14:02:50.496495 16178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:02:50.499439 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 16178
I20240501 14:02:50.499601 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:02:50.502609 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:0
--local_ip_for_outbound_sockets=127.22.33.70
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:50.510084 16305 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:50.510393 16305 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.511135 16305 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:50.512550 15595 ts_manager.cc:194] Registered new tserver with Master: acf31b7bd4974dc4951935d1c2da8f26 (127.22.33.69:37923)
I20240501 14:02:50.513644 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:45653
I20240501 14:02:50.571875 16045 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
W20240501 14:02:50.842890 16308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:50.843190 16308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:50.843276 16308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:50.850872 16308 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:50.851011 16308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:50.851130 16308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 14:02:50.860213 16308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:50.861588 16308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:50.863298 16308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:50.875695 16313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.876278 16316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.876659 16314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:50.877586 16308 server_base.cc:1034] running on GCE node
I20240501 14:02:50.878264 16308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:50.879225 16308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:50.880439 16308 hybrid_clock.cc:648] HybridClock initialized: now 1714572170880378 us; error 49 us; skew 500 ppm
I20240501 14:02:50.880820 16308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:50.883317 16308 webserver.cc:460] Webserver started at http://127.22.33.70:43099/ using document root <none> and password file <none>
I20240501 14:02:50.884114 16308 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:50.884228 16308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:50.884608 16308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:50.887444 16308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "bc2fcc831acb4607b85e5e6c328182e4"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.888263 16308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "bc2fcc831acb4607b85e5e6c328182e4"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.893540 16308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:02:50.896857 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.898347 16308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:50.898516 16308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bc2fcc831acb4607b85e5e6c328182e4"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:02:50.898728 16308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:50.918771 16308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:50.919659 16308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:50.919939 16308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:50.920889 16308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:50.922957 16308 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:50.923173 16308 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.923410 16308 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:50.923581 16308 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.957607 16308 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:44387
I20240501 14:02:50.957617 16434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:44387 every 8 connection(s)
I20240501 14:02:50.959028 16308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:02:50.968055 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 16308
I20240501 14:02:50.968259 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:02:50.971549 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:0
--local_ip_for_outbound_sockets=127.22.33.71
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:50.975240 16435 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:50.975570 16435 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.976374 16435 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:50.977794 15595 ts_manager.cc:194] Registered new tserver with Master: bc2fcc831acb4607b85e5e6c328182e4 (127.22.33.70:44387)
I20240501 14:02:50.978626 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:44015
I20240501 14:02:51.059794 16175 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
W20240501 14:02:51.312750 16438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:51.313122 16438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:51.313230 16438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:51.320667 16438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:51.320817 16438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:51.320942 16438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 14:02:51.330220 16438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:51.331566 16438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:51.333341 16438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:51.345861 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:51.346748 16438 server_base.cc:1034] running on GCE node
W20240501 14:02:51.347201 16444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:51.347710 16446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:51.348384 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:51.349428 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:51.350627 16438 hybrid_clock.cc:648] HybridClock initialized: now 1714572171350565 us; error 57 us; skew 500 ppm
I20240501 14:02:51.350950 16438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:51.353199 16438 webserver.cc:460] Webserver started at http://127.22.33.71:33645/ using document root <none> and password file <none>
I20240501 14:02:51.353950 16438 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:51.354072 16438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:51.354388 16438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:51.357000 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c747f91e33d9460f92d1fd00f12b4bba"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.357786 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c747f91e33d9460f92d1fd00f12b4bba"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.362941 16438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:51.366377 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.367720 16438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:51.367882 16438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c747f91e33d9460f92d1fd00f12b4bba"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.368101 16438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:51.400722 16438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:51.401726 16438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:51.402014 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:51.403106 16438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:51.404861 16438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:51.404963 16438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.405107 16438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:51.405179 16438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.437237 16438 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:39595
I20240501 14:02:51.437315 16564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:39595 every 8 connection(s)
I20240501 14:02:51.438587 16438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:02:51.445381 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 16438
I20240501 14:02:51.445541 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:02:51.448946 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.72:0
--local_ip_for_outbound_sockets=127.22.33.72
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:51.451387 16565 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:51.451704 16565 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:51.452569 16565 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:51.456553 15595 ts_manager.cc:194] Registered new tserver with Master: c747f91e33d9460f92d1fd00f12b4bba (127.22.33.71:39595)
I20240501 14:02:51.457403 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:43407
I20240501 14:02:51.516235 16305 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
W20240501 14:02:51.788619 16568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:51.788925 16568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:51.789081 16568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:51.796499 16568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:51.796612 16568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:51.796762 16568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.72
I20240501 14:02:51.805842 16568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.33.72
--webserver_port=0
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.72
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:51.807133 16568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:51.808779 16568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:51.820928 16573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:51.821694 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:51.822281 16576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:51.823007 16568 server_base.cc:1034] running on GCE node
I20240501 14:02:51.823585 16568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:51.824491 16568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:51.825690 16568 hybrid_clock.cc:648] HybridClock initialized: now 1714572171825646 us; error 39 us; skew 500 ppm
I20240501 14:02:51.826004 16568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:51.828205 16568 webserver.cc:460] Webserver started at http://127.22.33.72:39113/ using document root <none> and password file <none>
I20240501 14:02:51.828946 16568 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:51.829105 16568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:51.829420 16568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:51.832068 16568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "340d8cf614aa438c94feebe2ddddc00f"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.832821 16568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "340d8cf614aa438c94feebe2ddddc00f"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.838043 16568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:51.841404 16582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.842730 16568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:51.842890 16568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "340d8cf614aa438c94feebe2ddddc00f"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:02:51.843075 16568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:51.863063 16568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:51.863916 16568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:51.864183 16568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:51.865157 16568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:51.866619 16568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:51.866719 16568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.866830 16568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:51.866904 16568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:51.898551 16568 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.72:35717
I20240501 14:02:51.898603 16694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.72:35717 every 8 connection(s)
I20240501 14:02:51.899835 16568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:02:51.903352 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 16568
I20240501 14:02:51.903628 22661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:02:51.912451 16695 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:02:51.912739 16695 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:51.913489 16695 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:02:51.914757 15595 ts_manager.cc:194] Registered new tserver with Master: 340d8cf614aa438c94feebe2ddddc00f (127.22.33.72:35717)
I20240501 14:02:51.915509 15595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.72:43943
I20240501 14:02:51.919471 22661 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:02:51.935633 22661 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:02:51.935760 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 16178
I20240501 14:02:51.943675 16300 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:51.981210 16435 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
I20240501 14:02:52.026962 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 16178
I20240501 14:02:52.043288 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 16308
I20240501 14:02:52.051831 16430 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:52.132797 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 16308
I20240501 14:02:52.149806 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 16438
I20240501 14:02:52.157897 16560 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:52.237056 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 16438
I20240501 14:02:52.253540 22661 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskF0nCis/build/asan/bin/kudu pid 16568
I20240501 14:02:52.261669 16690 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:52.342895 22661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF0nCis/build/asan/bin/kudu with pid 16568
I20240501 14:02:56.111359 15595 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:56.161209 15849 tablet_service.cc:1467] Processing CreateTablet for tablet 1a762b8c33a143ac93a2263f92d5b714 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.162350 15848 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8331fc2f5f48278928267c260356c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.163436 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8331fc2f5f48278928267c260356c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.166311 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef97d453b774b988f34151eb73eee78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.166913 15980 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8331fc2f5f48278928267c260356c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.166913 16109 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef97d453b774b988f34151eb73eee78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.167709 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef97d453b774b988f34151eb73eee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.168547 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8331fc2f5f48278928267c260356c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.168563 16109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef97d453b774b988f34151eb73eee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.169543 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 1a762b8c33a143ac93a2263f92d5b714 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.170403 15717 tablet_service.cc:1467] Processing CreateTablet for tablet 1a762b8c33a143ac93a2263f92d5b714 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.170835 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a762b8c33a143ac93a2263f92d5b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.173352 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8331fc2f5f48278928267c260356c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.173795 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8331fc2f5f48278928267c260356c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.174811 15979 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef97d453b774b988f34151eb73eee78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6b1deddbb91b47c0bb05379e30657ba7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.175318 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef97d453b774b988f34151eb73eee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.177264 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a762b8c33a143ac93a2263f92d5b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.183513 16724 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:56.189389 16724 tablet_bootstrap.cc:654] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.189608 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a762b8c33a143ac93a2263f92d5b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.190969 16724 log.cc:824] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:56.191815 16726 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:56.193615 16725 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:56.198024 16726 tablet_bootstrap.cc:654] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.198109 16725 tablet_bootstrap.cc:654] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.199450 16726 log.cc:824] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:56.199453 16725 log.cc:824] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:56.200349 16724 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:56.200699 16724 ts_tablet_manager.cc:1367] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240501 14:02:56.209904 16724 raft_consensus.cc:348] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.210558 16724 raft_consensus.cc:374] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.210741 16724 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.211565 16724 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.215525 16726 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:56.215965 16726 ts_tablet_manager.cc:1367] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.009s
I20240501 14:02:56.217906 16730 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:56.218295 16725 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:56.218518 16724 ts_tablet_manager.cc:1398] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.018s user 0.008s sys 0.003s
I20240501 14:02:56.218695 16725 ts_tablet_manager.cc:1367] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.007s
I20240501 14:02:56.218964 16724 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:56.220678 16724 tablet_bootstrap.cc:654] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.224581 16724 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:56.224776 16724 ts_tablet_manager.cc:1367] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:02:56.225829 16724 raft_consensus.cc:348] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.226095 16724 raft_consensus.cc:374] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.226127 16730 tablet_bootstrap.cc:654] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.226214 16724 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.226472 16724 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.227046 16724 ts_tablet_manager.cc:1398] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:56.228349 16730 log.cc:824] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:56.228602 16725 raft_consensus.cc:348] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.229202 16725 raft_consensus.cc:374] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.229337 16725 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.230245 16725 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.231739 16726 raft_consensus.cc:348] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.232089 16726 raft_consensus.cc:374] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.232228 16726 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.232364 16730 tablet_bootstrap.cc:492] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:56.232704 16730 ts_tablet_manager.cc:1367] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240501 14:02:56.232909 16726 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.234269 16725 ts_tablet_manager.cc:1398] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
I20240501 14:02:56.234633 16726 ts_tablet_manager.cc:1398] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.018s user 0.011s sys 0.000s
I20240501 14:02:56.234717 16725 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:56.235047 16726 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:56.236814 16726 tablet_bootstrap.cc:654] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.237422 16735 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:56.237751 16725 tablet_bootstrap.cc:654] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.237730 16735 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.245858 16726 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:56.246070 16726 ts_tablet_manager.cc:1367] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 14:02:56.247457 16726 raft_consensus.cc:348] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.247749 16726 raft_consensus.cc:374] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.247470 16730 raft_consensus.cc:348] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.247871 16726 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.248045 16730 raft_consensus.cc:374] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.248220 16730 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.248150 16726 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.249369 16730 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.250924 16735 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:56.257349 16725 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:56.257700 16725 ts_tablet_manager.cc:1367] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.011s
I20240501 14:02:56.259157 16725 raft_consensus.cc:348] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.259548 16725 raft_consensus.cc:374] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.259703 16725 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.260133 16725 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.260903 16725 ts_tablet_manager.cc:1398] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:56.262928 16726 ts_tablet_manager.cc:1398] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20240501 14:02:56.264878 16730 ts_tablet_manager.cc:1398] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.032s user 0.015s sys 0.011s
I20240501 14:02:56.266286 16730 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:56.268534 16730 tablet_bootstrap.cc:654] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.269846 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:56.270478 15737 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:56.271325 15935 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:56.271862 16740 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:56.271927 16730 tablet_bootstrap.cc:492] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:56.271991 16740 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:56.271811 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:56.272063 16730 ts_tablet_manager.cc:1367] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:56.272146 16740 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.272384 15869 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:56.273116 16730 raft_consensus.cc:348] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.273309 16730 raft_consensus.cc:374] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.273402 16730 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.273612 16730 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.274044 16730 ts_tablet_manager.cc:1398] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:56.274374 16730 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:56.275004 16740 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.275535 16730 tablet_bootstrap.cc:654] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.275898 16740 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:56.276374 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:56.276371 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:56.276564 15737 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.276603 15869 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.279448 16730 tablet_bootstrap.cc:492] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:56.279503 15737 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:56.279606 16730 ts_tablet_manager.cc:1367] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:02:56.279779 15869 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:56.280292 15936 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:56.280378 16730 raft_consensus.cc:348] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.280627 16730 raft_consensus.cc:374] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.280652 16740 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:56.280709 16730 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.280864 16740 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:02:56.280965 16730 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.281412 16740 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.281666 16730 ts_tablet_manager.cc:1398] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:56.285653 15594 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
W20240501 14:02:56.291316 16176 tablet.cc:2369] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:56.298591 16734 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:56.298799 16734 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.300392 16734 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:56.307729 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:02:56.308218 16000 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:02:56.308779 16065 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:56.309193 16734 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:56.309334 16734 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:56.309444 16734 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.309613 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:56.309801 15737 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
W20240501 14:02:56.310348 16046 tablet.cc:2369] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:56.310750 16741 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:56.310912 16741 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.312152 16734 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.312676 16741 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:02:56.312820 16734 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:56.314271 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:02:56.314527 16000 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.315281 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:56.315510 15737 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.318333 15737 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:02:56.318830 16065 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:56.319157 16734 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:56.319415 16000 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:02:56.319661 16734 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:02:56.320274 16734 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:02:56.324138 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:56.324386 15869 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:56.325012 15673 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:56.325084 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:56.325446 16741 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:56.325459 16130 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:56.325579 16741 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:56.325698 16741 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.327436 15594 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 1 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:02:56.328114 16741 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.328919 16741 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:02:56.329249 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:56.329491 15869 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.329566 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:56.329754 16130 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:56.331416 15869 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:02:56.331820 15673 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:56.332049 16130 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:02:56.332192 16741 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:56.332733 16741 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:02:56.333376 16741 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:02:56.337818 15594 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
W20240501 14:02:56.369273 15915 tablet.cc:2369] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:56.399145 15783 tablet.cc:2369] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:56.686540 16753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:56.686854 16753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:56.694556 16753 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:56.717461 16740 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.725085 16735 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.728371 16751 consensus_queue.cc:1035] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.729447 16734 consensus_queue.cc:1035] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:02:56.738744 16751 consensus_queue.cc:1035] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.739301 16734 consensus_queue.cc:1035] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.088 ms
time per row: 0.115044 ms
I20240501 14:02:57.098627 15594 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:57.107704 16110 tablet_service.cc:1467] Processing CreateTablet for tablet c73309c7bc4644aa972bd08799670ef7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.107944 15849 tablet_service.cc:1467] Processing CreateTablet for tablet c73309c7bc4644aa972bd08799670ef7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.108222 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73309c7bc4644aa972bd08799670ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.108341 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73309c7bc4644aa972bd08799670ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.109408 15848 tablet_service.cc:1467] Processing CreateTablet for tablet b7d528abe1604862acc3bffe7fbb35a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.109427 15979 tablet_service.cc:1467] Processing CreateTablet for tablet c73309c7bc4644aa972bd08799670ef7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.109759 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d528abe1604862acc3bffe7fbb35a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.109877 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73309c7bc4644aa972bd08799670ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.109967 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 9434887fc6944dd587f54b28fbe73447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.110373 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434887fc6944dd587f54b28fbe73447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.113502 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 9434887fc6944dd587f54b28fbe73447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.113940 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434887fc6944dd587f54b28fbe73447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.116622 15980 tablet_service.cc:1467] Processing CreateTablet for tablet b7d528abe1604862acc3bffe7fbb35a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.117480 15978 tablet_service.cc:1467] Processing CreateTablet for tablet 9434887fc6944dd587f54b28fbe73447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.117882 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d528abe1604862acc3bffe7fbb35a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.121461 16109 tablet_service.cc:1467] Processing CreateTablet for tablet b7d528abe1604862acc3bffe7fbb35a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=416d3a63852141b5b7b3e53c460208c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:57.121915 16109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d528abe1604862acc3bffe7fbb35a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.122104 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434887fc6944dd587f54b28fbe73447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:57.134390 16815 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:57.136401 16817 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:57.136852 16815 tablet_bootstrap.cc:654] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.137064 16814 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:57.138999 16817 tablet_bootstrap.cc:654] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.138998 16814 tablet_bootstrap.cc:654] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.144394 16817 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:57.144522 16815 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:57.144644 16817 ts_tablet_manager.cc:1367] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:02:57.144752 16815 ts_tablet_manager.cc:1367] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240501 14:02:57.145583 16814 tablet_bootstrap.cc:492] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:57.145779 16814 ts_tablet_manager.cc:1367] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 14:02:57.145905 16817 raft_consensus.cc:348] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.146075 16815 raft_consensus.cc:348] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.146281 16817 raft_consensus.cc:374] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.146361 16815 raft_consensus.cc:374] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.146451 16817 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.146528 16815 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.146726 16814 raft_consensus.cc:348] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.146816 16815 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.146968 16814 raft_consensus.cc:374] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.147014 16817 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.147434 16815 ts_tablet_manager.cc:1398] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:57.147516 16816 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:57.147060 16814 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.148276 16815 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:57.148233 16814 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.149365 16814 ts_tablet_manager.cc:1398] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:57.149704 16814 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:57.149925 16815 tablet_bootstrap.cc:654] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.147990 16817 ts_tablet_manager.cc:1398] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:57.152233 16814 tablet_bootstrap.cc:654] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.155043 16817 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:57.156502 16816 tablet_bootstrap.cc:654] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.156759 16817 tablet_bootstrap.cc:654] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.157737 16814 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:57.158046 16814 ts_tablet_manager.cc:1367] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240501 14:02:57.159133 16814 raft_consensus.cc:348] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.159525 16814 raft_consensus.cc:374] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.159664 16814 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.160079 16814 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.160902 16814 ts_tablet_manager.cc:1398] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:57.162699 16815 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:57.163002 16815 ts_tablet_manager.cc:1367] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240501 14:02:57.163527 16817 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:57.163981 16815 raft_consensus.cc:348] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.164203 16815 raft_consensus.cc:374] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.164278 16815 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.164465 16816 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:57.164608 16816 ts_tablet_manager.cc:1367] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 14:02:57.164525 16815 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.164877 16817 ts_tablet_manager.cc:1367] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20240501 14:02:57.165212 16815 ts_tablet_manager.cc:1398] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:57.165530 16815 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:57.165472 16816 raft_consensus.cc:348] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.165748 16816 raft_consensus.cc:374] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.165887 16816 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.166043 16817 raft_consensus.cc:348] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.166316 16817 raft_consensus.cc:374] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.166183 16816 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.166421 16817 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.166975 16816 ts_tablet_manager.cc:1398] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:57.167161 16815 tablet_bootstrap.cc:654] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.167261 16817 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.170174 16815 tablet_bootstrap.cc:492] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:57.170280 16749 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.170336 16815 ts_tablet_manager.cc:1367] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:57.170382 16817 ts_tablet_manager.cc:1398] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:02:57.170439 16749 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.170696 16817 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:57.171116 16815 raft_consensus.cc:348] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.171311 16815 raft_consensus.cc:374] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.171316 16749 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:02:57.171408 16815 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.171615 16815 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.172163 16815 ts_tablet_manager.cc:1398] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:57.172370 16817 tablet_bootstrap.cc:654] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.172653 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:02:57.172856 16000 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:02:57.173683 16065 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:57.174055 16749 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.174163 16749 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.174257 16749 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.174934 16817 tablet_bootstrap.cc:492] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:57.175061 16817 ts_tablet_manager.cc:1367] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:02:57.175758 16817 raft_consensus.cc:348] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.175963 16817 raft_consensus.cc:374] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.176048 16817 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.176224 16817 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.176729 16817 ts_tablet_manager.cc:1398] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:57.178535 16749 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.182540 16749 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:02:57.183534 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:02:57.183776 16000 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.186374 16000 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:02:57.186774 16065 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:57.187081 16749 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.187243 16749 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:02:57.187542 16749 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.187824 15868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:57.187970 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:57.188017 15868 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:02:57.191068 15594 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 1 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:02:57.230432 16809 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.230607 16809 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.231528 16809 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:02:57.231986 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:57.232249 15869 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:57.232671 15936 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:57.233018 16809 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.233198 16809 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.233316 16809 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.236095 16809 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.236745 16809 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:02:57.237267 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:57.237520 15869 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.239253 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:57.239490 16130 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:57.239435 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:57.239766 16129 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.240310 15869 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:57.240757 15936 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:57.241086 16809 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.241240 16809 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:02:57.241489 16809 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:57.242262 16129 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:57.244300 15593 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:02:57.529335 16751 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.529528 16751 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.530336 16751 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:57.531150 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:57.531404 15869 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:57.531929 15673 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:57.532263 16751 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.532366 16751 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.532464 16751 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.534317 16751 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.534894 16751 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:57.535485 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:57.535701 15869 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.536816 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:02:57.536887 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:02:57.537084 16000 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:57.538132 15869 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:02:57.538476 15673 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:02:57.538754 16751 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.539094 16751 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:02:57.539347 16751 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:57.541837 15593 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:02:57.666007 16749 consensus_queue.cc:1035] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.670429 15869 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.674238 16773 consensus_queue.cc:1035] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.690114 16809 consensus_queue.cc:1035] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.694574 16809 consensus_queue.cc:1035] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:57.929363 16836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:57.929803 16836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:57.937377 16836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:02:58.030494 16751 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:58.037914 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.041680 16751 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 277.133 ms
time per row: 0.138567 ms
I20240501 14:02:58.367388 15593 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:58.376215 16110 tablet_service.cc:1467] Processing CreateTablet for tablet e8d8635d7c254bc5ae37c535e66b6dc9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.376571 15716 tablet_service.cc:1467] Processing CreateTablet for tablet e8d8635d7c254bc5ae37c535e66b6dc9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.376729 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d8635d7c254bc5ae37c535e66b6dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.377247 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d8635d7c254bc5ae37c535e66b6dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.377501 15849 tablet_service.cc:1467] Processing CreateTablet for tablet e8d8635d7c254bc5ae37c535e66b6dc9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.377898 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d8635d7c254bc5ae37c535e66b6dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.378060 15715 tablet_service.cc:1467] Processing CreateTablet for tablet e082a4dae96a481289b923efe77fa87b (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.378443 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e082a4dae96a481289b923efe77fa87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.381748 15717 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa0651edf94361bcc1efdcd6a2c7bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.382225 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa0651edf94361bcc1efdcd6a2c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.382498 16109 tablet_service.cc:1467] Processing CreateTablet for tablet e082a4dae96a481289b923efe77fa87b (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.382869 16109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e082a4dae96a481289b923efe77fa87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.385430 15979 tablet_service.cc:1467] Processing CreateTablet for tablet e082a4dae96a481289b923efe77fa87b (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.385843 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e082a4dae96a481289b923efe77fa87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.388640 16108 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa0651edf94361bcc1efdcd6a2c7bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.389153 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa0651edf94361bcc1efdcd6a2c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.393889 15848 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa0651edf94361bcc1efdcd6a2c7bb (DEFAULT_TABLE table=rebalance_test_table_2 [id=475e71be055345dc89bb327abb3bd253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:58.394448 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa0651edf94361bcc1efdcd6a2c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.400565 16907 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:58.403318 16907 tablet_bootstrap.cc:654] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.410537 16906 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:58.412416 16909 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:58.412698 16906 tablet_bootstrap.cc:654] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.414321 16909 tablet_bootstrap.cc:654] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.415159 16907 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:58.415383 16907 ts_tablet_manager.cc:1367] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240501 14:02:58.415578 16908 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:58.416321 16907 raft_consensus.cc:348] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.416671 16907 raft_consensus.cc:374] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.416810 16907 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.417194 16907 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.417547 16908 tablet_bootstrap.cc:654] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.417982 16906 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:58.418010 16907 ts_tablet_manager.cc:1398] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:58.418190 16906 ts_tablet_manager.cc:1367] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:02:58.419342 16906 raft_consensus.cc:348] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.419713 16906 raft_consensus.cc:374] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.419889 16906 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.420161 16906 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.421418 16909 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:58.421579 16909 ts_tablet_manager.cc:1367] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:02:58.421676 16906 ts_tablet_manager.cc:1398] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:58.422014 16906 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:58.422542 16909 raft_consensus.cc:348] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.422839 16909 raft_consensus.cc:374] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.422986 16909 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.423337 16909 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.423712 16906 tablet_bootstrap.cc:654] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.424790 16909 ts_tablet_manager.cc:1398] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:58.425181 16909 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:58.427325 16909 tablet_bootstrap.cc:654] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.431947 16906 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:58.432191 16906 ts_tablet_manager.cc:1367] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 14:02:58.433115 16906 raft_consensus.cc:348] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.433306 16906 raft_consensus.cc:374] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.433399 16906 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.433663 16906 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.434252 16906 ts_tablet_manager.cc:1398] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:58.434916 16906 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:58.435544 16908 tablet_bootstrap.cc:492] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:58.435818 16908 ts_tablet_manager.cc:1367] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.004s
I20240501 14:02:58.436684 16906 tablet_bootstrap.cc:654] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.437037 16908 raft_consensus.cc:348] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.437273 16908 raft_consensus.cc:374] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.437391 16908 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.437670 16908 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.438475 16908 ts_tablet_manager.cc:1398] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:58.438932 16908 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:58.440266 16909 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:58.440507 16909 ts_tablet_manager.cc:1367] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240501 14:02:58.440570 16908 tablet_bootstrap.cc:654] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.441414 16909 raft_consensus.cc:348] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.442667 16906 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:58.442811 16909 raft_consensus.cc:374] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.442834 16906 ts_tablet_manager.cc:1367] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240501 14:02:58.442979 16909 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.443215 16908 tablet_bootstrap.cc:492] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:58.443418 16908 ts_tablet_manager.cc:1367] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:02:58.443609 16906 raft_consensus.cc:348] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.443351 16909 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.443780 16906 raft_consensus.cc:374] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.443862 16906 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.444371 16909 ts_tablet_manager.cc:1398] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:58.444236 16908 raft_consensus.cc:348] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.444473 16908 raft_consensus.cc:374] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.444584 16908 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.444890 16906 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.444891 16908 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.445547 16908 ts_tablet_manager.cc:1398] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:58.445885 16908 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:58.446377 16906 ts_tablet_manager.cc:1398] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:58.447245 16908 tablet_bootstrap.cc:654] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:58.449831 16908 tablet_bootstrap.cc:492] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:58.450011 16908 ts_tablet_manager.cc:1367] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:02:58.450120 16894 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:58.450284 16894 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.450871 16908 raft_consensus.cc:348] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.450979 16894 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:58.451092 16908 raft_consensus.cc:374] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:58.451181 16908 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.451393 16908 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.451488 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:58.451696 15737 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:02:58.451670 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:58.451815 16908 ts_tablet_manager.cc:1398] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:58.451877 15867 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:02:58.452188 16065 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:58.452636 16894 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:58.452755 16894 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:58.452855 16894 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.455129 16894 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.455605 16894 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:58.456007 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:58.456022 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:58.456169 15737 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.456185 15867 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.458070 15737 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:02:58.458096 15867 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:02:58.458456 16065 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:58.458760 16894 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:58.458917 16894 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:02:58.459185 16894 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.461851 15592 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 1 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:02:58.482193 16868 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:58.482393 16868 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.483076 16868 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:58.483528 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:58.483573 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:58.483790 15867 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:58.483790 16129 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:02:58.484200 15673 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:02:58.484457 16868 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:58.484551 16868 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:58.484616 16868 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.486487 16868 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.486985 16868 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:58.487377 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:58.487499 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:58.487607 16129 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.487725 15867 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.490480 15867 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:02:58.490480 16129 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:02:58.490950 15671 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:58.491272 16868 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:58.491430 16868 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:02:58.491734 16868 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:58.494577 15592 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:02:58.558598 16809 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:58.558843 16809 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.559522 16809 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:58.560113 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:58.560366 15737 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:58.560460 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:58.560683 16129 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:58.560832 15935 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:58.561276 16809 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:58.561375 16809 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:58.561489 16809 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.564589 16809 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.565747 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:58.565757 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:58.565960 16129 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.565994 15737 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:58.567914 15737 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:58.568197 16809 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:58.568432 15935 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:58.568711 16809 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:58.568738 16129 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:58.568884 16809 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:02:58.569216 16809 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:58.572449 15592 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:02:58.901134 16894 consensus_queue.cc:1035] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:58.907320 16894 consensus_queue.cc:1035] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:58.936959 16922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:58.937774 16922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:58.945825 16922 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:58.975587 16868 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 14:02:58.980484 16868 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:58.988147 16809 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:58.999465 16735 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 288.308 ms
time per row: 0.144154 ms
I20240501 14:02:59.392639 15588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:59.401863 15979 tablet_service.cc:1467] Processing CreateTablet for tablet dc17177ae8b144c0ac3483d8a3540456 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.402122 15716 tablet_service.cc:1467] Processing CreateTablet for tablet dc17177ae8b144c0ac3483d8a3540456 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.402400 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17177ae8b144c0ac3483d8a3540456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.402526 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17177ae8b144c0ac3483d8a3540456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.402906 15849 tablet_service.cc:1467] Processing CreateTablet for tablet dc17177ae8b144c0ac3483d8a3540456 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.403230 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17177ae8b144c0ac3483d8a3540456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.405314 15848 tablet_service.cc:1467] Processing CreateTablet for tablet 9154ac5f942c449686c338594b42ee66 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.405669 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154ac5f942c449686c338594b42ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.405861 16110 tablet_service.cc:1467] Processing CreateTablet for tablet db5e643c5ec54233a680726986053d4f (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.406262 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5e643c5ec54233a680726986053d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.409495 16108 tablet_service.cc:1467] Processing CreateTablet for tablet 9154ac5f942c449686c338594b42ee66 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.410025 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154ac5f942c449686c338594b42ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.410310 15717 tablet_service.cc:1467] Processing CreateTablet for tablet db5e643c5ec54233a680726986053d4f (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.410815 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5e643c5ec54233a680726986053d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.414252 15978 tablet_service.cc:1467] Processing CreateTablet for tablet db5e643c5ec54233a680726986053d4f (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.414803 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5e643c5ec54233a680726986053d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.417476 15980 tablet_service.cc:1467] Processing CreateTablet for tablet 9154ac5f942c449686c338594b42ee66 (DEFAULT_TABLE table=rebalance_test_table_3 [id=60c2ff3bebfb443aa39aaff8780ed8f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.417901 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154ac5f942c449686c338594b42ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.418102 16983 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:59.419940 16983 tablet_bootstrap.cc:654] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.425212 16983 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:59.425417 16983 ts_tablet_manager.cc:1367] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 14:02:59.426296 16983 raft_consensus.cc:348] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.426551 16983 raft_consensus.cc:374] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.426687 16983 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.427028 16983 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.428303 16983 ts_tablet_manager.cc:1398] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:59.428580 16983 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:02:59.430871 16983 tablet_bootstrap.cc:654] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.432929 16987 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:59.435509 16987 tablet_bootstrap.cc:654] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.435580 16983 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:02:59.435773 16983 ts_tablet_manager.cc:1367] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 14:02:59.436657 16983 raft_consensus.cc:348] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.437077 16983 raft_consensus.cc:374] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.437671 16983 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.438038 16983 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.440500 16983 ts_tablet_manager.cc:1398] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:59.440575 16985 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:59.442909 16985 tablet_bootstrap.cc:654] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.444155 16987 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:59.444337 16987 ts_tablet_manager.cc:1367] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:02:59.444576 16982 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:59.445382 16985 tablet_bootstrap.cc:492] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:59.445346 16987 raft_consensus.cc:348] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.445693 16987 raft_consensus.cc:374] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.445652 16985 ts_tablet_manager.cc:1367] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:59.445842 16987 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.446120 16987 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.446424 16982 tablet_bootstrap.cc:654] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.446758 16987 ts_tablet_manager.cc:1398] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:59.447033 16987 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:02:59.446996 16985 raft_consensus.cc:348] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.447319 16985 raft_consensus.cc:374] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.447491 16985 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.447875 16985 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.448912 16987 tablet_bootstrap.cc:654] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.449525 16985 ts_tablet_manager.cc:1398] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:59.449934 16985 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:59.452050 16985 tablet_bootstrap.cc:654] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.453974 16987 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:02:59.455919 16987 ts_tablet_manager.cc:1367] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240501 14:02:59.456559 16987 raft_consensus.cc:348] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.456750 16987 raft_consensus.cc:374] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.456831 16987 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.457101 16987 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.457720 16987 ts_tablet_manager.cc:1398] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:59.459010 16982 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:59.459167 16982 ts_tablet_manager.cc:1367] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240501 14:02:59.459270 16985 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:59.459457 16985 ts_tablet_manager.cc:1367] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 14:02:59.459909 16982 raft_consensus.cc:348] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.460072 16982 raft_consensus.cc:374] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.460178 16982 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.460199 16985 raft_consensus.cc:348] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.460449 16985 raft_consensus.cc:374] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.460536 16985 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.460464 16982 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.460785 16985 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.461517 16985 ts_tablet_manager.cc:1398] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:59.461773 16982 ts_tablet_manager.cc:1398] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:59.461814 16985 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:02:59.462050 16982 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:02:59.463466 16985 tablet_bootstrap.cc:654] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.463604 16982 tablet_bootstrap.cc:654] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.465312 16985 tablet_bootstrap.cc:492] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:02:59.465454 16982 tablet_bootstrap.cc:492] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:02:59.465453 16985 ts_tablet_manager.cc:1367] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:59.465621 16982 ts_tablet_manager.cc:1367] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:59.466044 16985 raft_consensus.cc:348] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.466262 16985 raft_consensus.cc:374] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.466363 16985 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.466571 16985 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.467032 16985 ts_tablet_manager.cc:1398] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:02:59.467011 16982 raft_consensus.cc:348] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.467244 16982 raft_consensus.cc:374] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.467324 16982 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.467554 16982 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.468261 16982 ts_tablet_manager.cc:1398] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:59.551139 16809 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.551365 16809 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.551949 16809 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:59.552588 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:59.552641 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:02:59.552790 15736 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:59.552873 15869 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:59.553285 15935 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:59.553560 16809 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.553660 16809 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.553784 16809 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.555758 16809 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.556427 16809 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:02:59.556896 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:02:59.556910 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:59.557117 15869 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.557132 15736 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.559126 15736 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:59.559542 15935 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:59.559791 16809 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.559877 15869 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:59.559971 16809 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:02:59.560225 16809 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.563300 15592 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:02:59.659418 16731 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.659629 16731 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.661484 16731 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:02:59.673808 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:59.674125 16130 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:02:59.674793 15803 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:59.675413 16731 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.675540 16731 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.675650 16731 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.678361 16731 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.679257 16731 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:02:59.679366 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:59.679577 16130 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.680848 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:02:59.680995 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:02:59.681142 16000 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:02:59.681830 16130 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:02:59.682252 15803 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:02:59.682552 16731 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.682941 16731 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:02:59.683475 16731 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:02:59.686586 15592 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 1 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:02:59.747115 16809 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.747370 16809 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.748042 16809 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:59.748653 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:02:59.748667 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:02:59.748895 16130 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:59.748914 15736 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:02:59.749375 15934 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:02:59.749627 16809 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.749735 16809 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.749827 16809 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.751772 16809 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.752382 16809 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:02:59.752873 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:02:59.752854 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:02:59.753103 16130 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.753122 15736 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.755887 15736 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:59.755906 16130 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:02:59.756321 15935 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:02:59.756609 16809 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.756769 16809 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:02:59.757076 16809 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:02:59.760682 15592 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:02:59.946915 16809 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:59.953059 16809 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.107101 16731 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.113669 15999 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.117798 16731 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:00.153990 17003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:00.154450 17003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:00.162165 17003 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:00.188050 16809 consensus_queue.cc:1035] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.192520 16809 consensus_queue.cc:1035] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 237.223 ms
time per row: 0.118612 ms
I20240501 14:03:00.548998 15587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:00.558265 15717 tablet_service.cc:1467] Processing CreateTablet for tablet 4b01b2fe30b043fdb721ead3019caeea (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.558681 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b01b2fe30b043fdb721ead3019caeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.559803 15978 tablet_service.cc:1467] Processing CreateTablet for tablet 4b01b2fe30b043fdb721ead3019caeea (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.559947 15716 tablet_service.cc:1467] Processing CreateTablet for tablet b99a2f1fb5d64858a323a0c2dd9eab5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.560012 15979 tablet_service.cc:1467] Processing CreateTablet for tablet 7042e84f9fdb4a7c81fb314d14a5d29e (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.560243 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b01b2fe30b043fdb721ead3019caeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.560396 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a2f1fb5d64858a323a0c2dd9eab5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.560689 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 7042e84f9fdb4a7c81fb314d14a5d29e (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.561064 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7042e84f9fdb4a7c81fb314d14a5d29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.562117 16108 tablet_service.cc:1467] Processing CreateTablet for tablet b99a2f1fb5d64858a323a0c2dd9eab5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.562528 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a2f1fb5d64858a323a0c2dd9eab5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.565377 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 7042e84f9fdb4a7c81fb314d14a5d29e (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.565758 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7042e84f9fdb4a7c81fb314d14a5d29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.566845 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7042e84f9fdb4a7c81fb314d14a5d29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.569912 15848 tablet_service.cc:1467] Processing CreateTablet for tablet b99a2f1fb5d64858a323a0c2dd9eab5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.570365 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a2f1fb5d64858a323a0c2dd9eab5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.575690 17070 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:00.576277 15849 tablet_service.cc:1467] Processing CreateTablet for tablet 4b01b2fe30b043fdb721ead3019caeea (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ce8f804d88476f85c04cbd8ca8cc12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.576680 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b01b2fe30b043fdb721ead3019caeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.577674 17070 tablet_bootstrap.cc:654] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.591624 17070 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:00.591802 17070 ts_tablet_manager.cc:1367] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240501 14:03:00.592617 17070 raft_consensus.cc:348] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.592832 17070 raft_consensus.cc:374] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.592931 17070 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.593268 17074 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:00.593222 17070 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.595304 17074 tablet_bootstrap.cc:654] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.595768 17072 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:00.596163 17070 ts_tablet_manager.cc:1398] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:03:00.596463 17070 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:00.597415 17071 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:00.598062 17070 tablet_bootstrap.cc:654] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.601325 17072 tablet_bootstrap.cc:654] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.602859 17071 tablet_bootstrap.cc:654] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.604598 17072 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:00.604743 17072 ts_tablet_manager.cc:1367] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:03:00.605675 17072 raft_consensus.cc:348] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.606036 17072 raft_consensus.cc:374] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.606200 17072 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.606537 17072 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.607502 17072 ts_tablet_manager.cc:1398] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:03:00.610258 17070 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:00.610391 17071 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:00.610450 17070 ts_tablet_manager.cc:1367] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20240501 14:03:00.610543 17071 ts_tablet_manager.cc:1367] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 14:03:00.610662 17074 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:00.610852 17074 ts_tablet_manager.cc:1367] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20240501 14:03:00.611269 17070 raft_consensus.cc:348] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.611335 17071 raft_consensus.cc:348] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.611519 17070 raft_consensus.cc:374] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.611567 17071 raft_consensus.cc:374] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.611606 17070 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.611711 17071 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.611889 17070 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.611984 17071 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.612515 17070 ts_tablet_manager.cc:1398] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:00.613340 17074 raft_consensus.cc:348] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.613497 17071 ts_tablet_manager.cc:1398] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:00.613648 17074 raft_consensus.cc:374] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.613759 17071 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:00.613816 17074 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.614148 17074 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.615451 17071 tablet_bootstrap.cc:654] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.616595 17074 ts_tablet_manager.cc:1398] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:03:00.616955 17074 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:00.619259 17074 tablet_bootstrap.cc:654] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.619522 17072 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:00.620870 17072 tablet_bootstrap.cc:654] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.621812 17074 tablet_bootstrap.cc:492] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:00.622000 17074 ts_tablet_manager.cc:1367] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:03:00.622130 17071 tablet_bootstrap.cc:492] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:00.622262 17071 ts_tablet_manager.cc:1367] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240501 14:03:00.622833 17074 raft_consensus.cc:348] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.623024 17074 raft_consensus.cc:374] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.623102 17074 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.623055 17071 raft_consensus.cc:348] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.623293 17071 raft_consensus.cc:374] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.623379 17071 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.623380 17074 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.623674 17071 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.623917 17074 ts_tablet_manager.cc:1398] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:03:00.622929 17072 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:00.624702 17071 ts_tablet_manager.cc:1398] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:03:00.625046 17071 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:00.626197 17072 ts_tablet_manager.cc:1367] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 14:03:00.626904 17071 tablet_bootstrap.cc:654] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:00.628809 17072 raft_consensus.cc:348] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.629242 17072 raft_consensus.cc:374] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.629395 17072 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.629743 17072 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.630486 17072 ts_tablet_manager.cc:1398] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:03:00.631669 17071 tablet_bootstrap.cc:492] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:00.631810 17071 ts_tablet_manager.cc:1367] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240501 14:03:00.632437 17071 raft_consensus.cc:348] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.632596 17071 raft_consensus.cc:374] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:00.632694 17071 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.632889 17071 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.633423 17071 ts_tablet_manager.cc:1398] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:00.652487 16834 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:00.652729 16834 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.653621 16834 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:00.654014 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:00.654261 15999 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:00.654390 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:00.654623 15672 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:00.654652 15867 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:00.654949 16834 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:00.655054 16834 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:00.655138 16834 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.658098 16834 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.658813 16834 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:00.659368 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:00.659422 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:00.659581 15999 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.659628 15867 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.661916 15867 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:00.662406 15673 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:00.662407 15999 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:00.662732 16834 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:00.663187 16834 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:00.663540 16834 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:00.666605 15587 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:00.692879 16834 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:00.693185 16834 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.693818 16834 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:00.694393 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:00.694473 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:00.694617 16130 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:00.694732 15999 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:00.695135 15671 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:00.695475 16834 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:00.695574 16834 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:00.695695 16834 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.698177 16834 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.698803 16834 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:00.699309 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:00.699288 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:00.699505 15999 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.699527 16130 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.702358 15999 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:00.702339 16130 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:00.702771 15672 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:00.703076 16834 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:00.703233 16834 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:00.703493 16834 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:00.706265 15587 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:00.766878 16731 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:00.767098 16731 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.767849 16731 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:00.768517 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:00.768885 16130 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:00.769357 15803 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:00.769688 16731 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:00.769814 16731 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:00.769938 16731 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.772527 16731 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.773248 16731 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:00.773854 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:00.774101 16130 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:00.775463 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:00.775592 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:00.775677 15737 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:00.776523 16130 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:00.776907 15803 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:00.777243 16731 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:00.777405 16731 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:00.777707 16731 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:00.780336 15587 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 1 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:01.128831 16834 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.145610 17083 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:01.174118 17089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:01.174440 17089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:03:01.181063 17083 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:01.182847 17089 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:01.185966 17083 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 14:03:01.229794 16731 consensus_queue.cc:1035] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.235009 15736 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.238754 17000 consensus_queue.cc:1035] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.997 ms
time per row: 0.133999 ms
I20240501 14:03:01.606273 15594 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:01.614915 15978 tablet_service.cc:1467] Processing CreateTablet for tablet 4a80377b51164fc5a0b6a6ed342488ea (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.615233 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 4a80377b51164fc5a0b6a6ed342488ea (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.615415 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a80377b51164fc5a0b6a6ed342488ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.615664 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a80377b51164fc5a0b6a6ed342488ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.616214 15848 tablet_service.cc:1467] Processing CreateTablet for tablet 4a80377b51164fc5a0b6a6ed342488ea (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.616663 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a80377b51164fc5a0b6a6ed342488ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.616815 15849 tablet_service.cc:1467] Processing CreateTablet for tablet da9d4827abaa478e9c63f779793ed0c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.617664 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 7c18622bc24146d99b2a779cc4e478fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.617765 15980 tablet_service.cc:1467] Processing CreateTablet for tablet 7c18622bc24146d99b2a779cc4e478fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.618175 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c18622bc24146d99b2a779cc4e478fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.621342 15979 tablet_service.cc:1467] Processing CreateTablet for tablet da9d4827abaa478e9c63f779793ed0c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.621744 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d4827abaa478e9c63f779793ed0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.623273 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d4827abaa478e9c63f779793ed0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.625188 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c18622bc24146d99b2a779cc4e478fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.625608 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 7c18622bc24146d99b2a779cc4e478fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.627490 16108 tablet_service.cc:1467] Processing CreateTablet for tablet da9d4827abaa478e9c63f779793ed0c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73db499b194d426c8fc6e73ef1fdeb08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.628006 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d4827abaa478e9c63f779793ed0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.630151 17159 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:01.631836 17159 tablet_bootstrap.cc:654] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.633159 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c18622bc24146d99b2a779cc4e478fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.636794 17160 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:01.640043 17160 tablet_bootstrap.cc:654] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.646867 17163 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:01.650633 17164 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:01.650789 17163 tablet_bootstrap.cc:654] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.652053 17160 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:01.652220 17160 ts_tablet_manager.cc:1367] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240501 14:03:01.653095 17160 raft_consensus.cc:348] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.653316 17160 raft_consensus.cc:374] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.653395 17160 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.653653 17160 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.654338 17160 ts_tablet_manager.cc:1398] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:01.654625 17160 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:01.660892 17163 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:01.661262 17163 ts_tablet_manager.cc:1367] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240501 14:03:01.662564 17163 raft_consensus.cc:348] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.663012 17163 raft_consensus.cc:374] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.663096 17160 tablet_bootstrap.cc:654] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.663208 17163 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.663545 17163 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.664342 17163 ts_tablet_manager.cc:1398] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:01.664718 17164 tablet_bootstrap.cc:654] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.669890 17159 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:01.670105 17159 ts_tablet_manager.cc:1367] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.016s
I20240501 14:03:01.670785 17160 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:01.670810 17159 raft_consensus.cc:348] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.670979 17160 ts_tablet_manager.cc:1367] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240501 14:03:01.671039 17159 raft_consensus.cc:374] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.671144 17159 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.671432 17164 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:01.671381 17159 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.671749 17164 ts_tablet_manager.cc:1367] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240501 14:03:01.671909 17159 ts_tablet_manager.cc:1398] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:01.671820 17160 raft_consensus.cc:348] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.672034 17160 raft_consensus.cc:374] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.672122 17160 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.672464 17159 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:01.672448 17160 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.673028 17160 ts_tablet_manager.cc:1398] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:03:01.673316 17160 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:01.673856 17159 tablet_bootstrap.cc:654] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.674901 17160 tablet_bootstrap.cc:654] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.677852 17159 tablet_bootstrap.cc:492] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:01.678086 17159 ts_tablet_manager.cc:1367] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:03:01.678941 17159 raft_consensus.cc:348] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.679241 17159 raft_consensus.cc:374] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.679342 17159 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.679617 17159 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.680246 17159 ts_tablet_manager.cc:1398] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:01.680557 17159 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:01.682348 17159 tablet_bootstrap.cc:654] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.683327 17164 raft_consensus.cc:348] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.686338 17164 raft_consensus.cc:374] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.686530 17164 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.686978 17164 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.691222 17164 ts_tablet_manager.cc:1398] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240501 14:03:01.696954 17164 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:01.699584 17164 tablet_bootstrap.cc:654] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.702219 17160 tablet_bootstrap.cc:492] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:01.702432 17164 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:01.702417 17160 ts_tablet_manager.cc:1367] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.007s
I20240501 14:03:01.702525 17159 tablet_bootstrap.cc:492] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:01.702692 17159 ts_tablet_manager.cc:1367] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.006s
I20240501 14:03:01.702697 17164 ts_tablet_manager.cc:1367] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:03:01.703279 17160 raft_consensus.cc:348] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.703491 17160 raft_consensus.cc:374] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.703406 17159 raft_consensus.cc:348] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.703588 17159 raft_consensus.cc:374] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.703593 17160 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.703691 17159 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.703732 17164 raft_consensus.cc:348] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.703877 17160 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.703914 17159 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.704027 17164 raft_consensus.cc:374] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.704280 17164 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.704514 17160 ts_tablet_manager.cc:1398] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:03:01.704639 17164 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.705017 17159 ts_tablet_manager.cc:1398] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:01.706754 17164 ts_tablet_manager.cc:1398] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:03:01.812937 16731 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.813225 16731 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.813911 16731 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:01.814532 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:01.814538 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:01.814757 15999 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:01.814787 16130 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:01.815287 15804 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:01.815634 16731 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.815732 16731 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.815835 16731 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.819036 16731 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.819805 16731 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:01.820313 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:01.820454 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:01.820539 15999 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.820695 16130 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.823612 16130 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:01.823611 15999 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:01.823943 16735 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.824072 15804 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:01.824155 16735 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.824446 16731 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.824667 16731 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:01.824937 16735 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:01.824993 16731 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:01.825656 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:01.825870 16130 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:01.826301 15934 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:01.826642 16735 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.826630 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:01.826740 16735 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.826853 16735 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.826982 15869 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:01.829025 15591 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 1 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:01.829639 16735 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.830344 16735 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:01.830914 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:01.831131 16130 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.831796 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:01.832019 15869 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.833529 16130 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:01.833985 15934 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:01.834302 16735 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.834728 16735 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:01.834987 16735 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:01.837895 15591 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:01.855063 15869 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:01.987516 17099 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.987728 17099 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.988556 17099 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:01.988538 16735 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.988731 16735 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.989106 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:01.989091 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:01.989327 15999 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:01.989323 15869 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:01.989765 15672 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:01.989998 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:01.990075 17099 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.990180 15869 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:01.990214 17099 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.990278 17099 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.990358 16735 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:01.990581 15936 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:01.991000 16735 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.991016 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:01.991127 16735 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.991261 16735 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.992623 17099 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.993005 15736 raft_consensus.cc:2379] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 in current term 1: Already voted for candidate 28461d3f709145e29fc3d3d1c7cce205 in this term.
I20240501 14:03:01.993217 16735 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.993310 17099 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:01.994022 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:01.994138 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:01.994146 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:01.994237 15869 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.994400 15999 raft_consensus.cc:2379] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 in current term 1: Already voted for candidate 09ee1930dbed4ac897d146bf885d0e87 in this term.
I20240501 14:03:01.996610 15869 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:01.996953 16735 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:01.997051 15673 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:01.997565 17099 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.997656 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:01.997857 17099 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:01.998101 17099 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:01.999852 15935 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:02.000154 16735 raft_consensus.cc:2726] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:03:02.001364 15591 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:02.269872 16735 consensus_queue.cc:1035] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.278951 17173 consensus_queue.cc:1035] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.331228 16731 consensus_queue.cc:1035] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.338141 17183 consensus_queue.cc:1035] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.441437 17135 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.446858 17135 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:02.453073 17176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:02.453441 17176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:02.461741 17176 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 249.482 ms
time per row: 0.124741 ms
I20240501 14:03:02.862437 15592 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:02.871404 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3735e79e334359b05c013508d99f3c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.871860 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3735e79e334359b05c013508d99f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.872089 16108 tablet_service.cc:1467] Processing CreateTablet for tablet ce980b7f336e48acb414191b25f92136 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.872464 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce980b7f336e48acb414191b25f92136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.873425 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3735e79e334359b05c013508d99f3c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.873816 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3735e79e334359b05c013508d99f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.876106 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3735e79e334359b05c013508d99f3c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.876550 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3735e79e334359b05c013508d99f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.878854 15715 tablet_service.cc:1467] Processing CreateTablet for tablet d773c0f7122f47fbbb79970f00ae137f (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.879237 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d773c0f7122f47fbbb79970f00ae137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.881534 15979 tablet_service.cc:1467] Processing CreateTablet for tablet ce980b7f336e48acb414191b25f92136 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.882077 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce980b7f336e48acb414191b25f92136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.885587 15717 tablet_service.cc:1467] Processing CreateTablet for tablet ce980b7f336e48acb414191b25f92136 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.886135 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce980b7f336e48acb414191b25f92136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.886373 17237 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:02.888568 17237 tablet_bootstrap.cc:654] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.890655 16110 tablet_service.cc:1467] Processing CreateTablet for tablet d773c0f7122f47fbbb79970f00ae137f (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.891095 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d773c0f7122f47fbbb79970f00ae137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.891624 15848 tablet_service.cc:1467] Processing CreateTablet for tablet d773c0f7122f47fbbb79970f00ae137f (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c5d1d9c6e754c5ebd0f4cdafdb4090a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:02.892150 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d773c0f7122f47fbbb79970f00ae137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:02.897823 17237 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:02.898061 17237 ts_tablet_manager.cc:1367] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240501 14:03:02.899009 17237 raft_consensus.cc:348] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.899369 17237 raft_consensus.cc:374] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.899511 17237 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.899832 17237 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.901957 17237 ts_tablet_manager.cc:1398] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:03:02.902288 17237 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:02.904008 17237 tablet_bootstrap.cc:654] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.911949 17240 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:02.915016 17236 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:02.915432 17241 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:02.915647 17240 tablet_bootstrap.cc:654] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.916870 17236 tablet_bootstrap.cc:654] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.917081 17237 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:02.917237 17237 ts_tablet_manager.cc:1367] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20240501 14:03:02.917569 17241 tablet_bootstrap.cc:654] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.918006 17237 raft_consensus.cc:348] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.918239 17237 raft_consensus.cc:374] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.918325 17237 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.918569 17237 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.919047 17237 ts_tablet_manager.cc:1398] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:02.919512 17237 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:02.921207 17237 tablet_bootstrap.cc:654] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.922130 17240 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:02.923172 17240 ts_tablet_manager.cc:1367] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:03:02.924094 17240 raft_consensus.cc:348] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.924332 17240 raft_consensus.cc:374] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.924440 17240 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.924670 17240 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.925271 17240 ts_tablet_manager.cc:1398] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:02.925660 17240 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:02.926426 17237 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:02.926582 17237 ts_tablet_manager.cc:1367] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:03:02.927364 17237 raft_consensus.cc:348] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.927563 17237 raft_consensus.cc:374] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.927660 17237 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.927999 17240 tablet_bootstrap.cc:654] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.927884 17237 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.929689 17237 ts_tablet_manager.cc:1398] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:03:02.934010 17241 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:02.934213 17241 ts_tablet_manager.cc:1367] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20240501 14:03:02.937302 17240 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:02.937549 17240 ts_tablet_manager.cc:1367] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:03:02.937799 17241 raft_consensus.cc:348] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.938130 17241 raft_consensus.cc:374] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.938251 17241 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.938359 17240 raft_consensus.cc:348] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.938611 17240 raft_consensus.cc:374] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.938694 17240 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.938588 17241 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.938961 17240 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.939512 17240 ts_tablet_manager.cc:1398] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:02.940857 17236 tablet_bootstrap.cc:492] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:02.940958 17241 ts_tablet_manager.cc:1398] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 14:03:02.941082 17236 ts_tablet_manager.cc:1367] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.010s
I20240501 14:03:02.941943 17236 raft_consensus.cc:348] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.942196 17236 raft_consensus.cc:374] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.942337 17236 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.942600 17236 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.943186 17236 ts_tablet_manager.cc:1398] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:02.943478 17236 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:02.945180 17236 tablet_bootstrap.cc:654] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.948246 17236 tablet_bootstrap.cc:492] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:02.948572 17236 ts_tablet_manager.cc:1367] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:03:02.949311 17236 raft_consensus.cc:348] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.949589 17236 raft_consensus.cc:374] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.949685 17236 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.949951 17236 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.950382 17236 ts_tablet_manager.cc:1398] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:02.950724 17236 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:02.951980 17236 tablet_bootstrap.cc:654] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:02.953824 17236 tablet_bootstrap.cc:492] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:02.954003 17236 ts_tablet_manager.cc:1367] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:03:02.954725 17236 raft_consensus.cc:348] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.954869 17236 raft_consensus.cc:374] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:02.954932 17236 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:02.955129 17236 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.955603 17236 ts_tablet_manager.cc:1398] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:03:02.967149 17099 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:02.967382 17099 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.968070 17099 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:02.968575 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:02.968714 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:02.968804 16129 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:02.969004 15869 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:02.969333 15671 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:02.969656 17099 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:02.969755 17099 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:02.969869 17099 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.971705 17099 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.972270 17099 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:02.972688 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:02.972690 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:02.972868 15869 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.972901 16129 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.974753 15869 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:02.974750 16129 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:02.975173 15673 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:02.975473 17099 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:02.975646 17099 raft_consensus.cc:686] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:02.975915 17099 consensus_queue.cc:237] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:02.978928 15593 catalog_manager.cc:5484] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:02.983933 16735 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:02.984097 16735 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.984697 16735 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:02.985200 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:02.985327 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:02.985459 15736 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:02.985582 16129 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:02.985885 15935 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:02.986315 16735 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:02.986438 16735 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:02.986536 16735 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.988647 16735 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.989231 16735 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:02.989704 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:02.989735 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:02.989923 16129 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.989935 15736 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:02.992856 15736 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:02.992868 16129 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:02.993288 15934 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:02.993585 16735 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:02.993796 16735 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:02.994494 16735 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:02.997493 15593 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:03.250221 16894 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:03.250461 16894 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:03.251212 16894 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:03.251719 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:03.251816 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:03.251978 15736 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:03:03.252060 15869 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:03:03.252498 16065 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:03.252780 16894 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:03.252862 16894 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:03.253131 16894 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:03.255084 16894 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:03.255695 16894 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:03.256266 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:03.256321 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:03.256470 15736 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:03.256537 15869 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:03.258879 15736 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:03:03.259085 15869 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:03:03.259430 16065 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:03.259750 16894 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:03.260093 16894 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:03.260393 16894 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:03.263370 15593 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 1 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:03.386749 17099 consensus_queue.cc:1035] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:03.392688 17135 consensus_queue.cc:1035] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:03.478848 16735 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:03.486456 17173 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:03.711874 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:03.712220 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:03.720837 17253 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:03:03.791700 16894 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:03.819249 16894 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 319.271 ms
time per row: 0.159636 ms
I20240501 14:03:04.197376 15586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:04.206409 15848 tablet_service.cc:1467] Processing CreateTablet for tablet f21aad1a0b1740a09ff3774295ad6928 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.206908 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21aad1a0b1740a09ff3774295ad6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.207109 15717 tablet_service.cc:1467] Processing CreateTablet for tablet f21aad1a0b1740a09ff3774295ad6928 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.207520 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21aad1a0b1740a09ff3774295ad6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.207923 15979 tablet_service.cc:1467] Processing CreateTablet for tablet f21aad1a0b1740a09ff3774295ad6928 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.208348 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21aad1a0b1740a09ff3774295ad6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.208427 16108 tablet_service.cc:1467] Processing CreateTablet for tablet 433aab52303a4ae5af6a65cc8a97481a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.208825 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433aab52303a4ae5af6a65cc8a97481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.213445 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 433aab52303a4ae5af6a65cc8a97481a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.213979 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433aab52303a4ae5af6a65cc8a97481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.216531 15715 tablet_service.cc:1467] Processing CreateTablet for tablet b2b6db3827b74bed8016ac9aef243527 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.217139 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b6db3827b74bed8016ac9aef243527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.217716 17327 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:04.217597 15978 tablet_service.cc:1467] Processing CreateTablet for tablet 433aab52303a4ae5af6a65cc8a97481a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.218132 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433aab52303a4ae5af6a65cc8a97481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.219628 17327 tablet_bootstrap.cc:654] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.221576 16110 tablet_service.cc:1467] Processing CreateTablet for tablet b2b6db3827b74bed8016ac9aef243527 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.222139 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b6db3827b74bed8016ac9aef243527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.222169 15980 tablet_service.cc:1467] Processing CreateTablet for tablet b2b6db3827b74bed8016ac9aef243527 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d0ccd4e4ef3b42be9ba80935177f24d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:04.222704 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b6db3827b74bed8016ac9aef243527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:04.245692 17329 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:04.247606 17325 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:04.248134 17329 tablet_bootstrap.cc:654] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.250111 17327 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:04.250363 17327 ts_tablet_manager.cc:1367] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.012s
I20240501 14:03:04.250546 17326 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:04.251433 17327 raft_consensus.cc:348] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.251711 17327 raft_consensus.cc:374] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.251855 17327 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.252156 17327 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.252671 17329 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:04.252808 17327 ts_tablet_manager.cc:1398] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:03:04.252825 17329 ts_tablet_manager.cc:1367] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:03:04.253193 17327 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:04.252386 17325 tablet_bootstrap.cc:654] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.253892 17326 tablet_bootstrap.cc:654] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.253857 17329 raft_consensus.cc:348] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.254099 17329 raft_consensus.cc:374] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.254215 17329 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.254451 17329 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.254951 17327 tablet_bootstrap.cc:654] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.255048 17329 ts_tablet_manager.cc:1398] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:04.257403 17329 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:04.259773 17329 tablet_bootstrap.cc:654] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.264854 17327 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:04.265380 17327 ts_tablet_manager.cc:1367] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240501 14:03:04.266481 17327 raft_consensus.cc:348] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.266775 17327 raft_consensus.cc:374] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.266918 17327 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.267290 17327 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.268328 17327 ts_tablet_manager.cc:1398] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:04.270779 17325 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:04.271138 17325 ts_tablet_manager.cc:1367] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240501 14:03:04.271718 17329 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:04.271900 17329 ts_tablet_manager.cc:1367] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20240501 14:03:04.272445 17325 raft_consensus.cc:348] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.272858 17325 raft_consensus.cc:374] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.273037 17325 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.273440 17329 raft_consensus.cc:348] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.273396 17325 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.273626 17329 raft_consensus.cc:374] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.273705 17329 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.274094 17325 ts_tablet_manager.cc:1398] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:04.274434 17325 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:04.274513 17329 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.275137 17329 ts_tablet_manager.cc:1398] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:03:04.275516 17326 tablet_bootstrap.cc:492] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:04.275918 17326 ts_tablet_manager.cc:1367] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.010s
I20240501 14:03:04.275974 17325 tablet_bootstrap.cc:654] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.276839 17326 raft_consensus.cc:348] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.277102 17326 raft_consensus.cc:374] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.277220 17326 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.277455 17326 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.277906 17325 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:04.278048 17326 ts_tablet_manager.cc:1398] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:04.278085 17325 ts_tablet_manager.cc:1367] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:03:04.278740 17326 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:04.279376 17325 raft_consensus.cc:348] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.279657 17325 raft_consensus.cc:374] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.279783 17325 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.280113 17325 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.280718 17325 ts_tablet_manager.cc:1398] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:04.282172 17326 tablet_bootstrap.cc:654] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.285120 17326 tablet_bootstrap.cc:492] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:04.285283 17326 ts_tablet_manager.cc:1367] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:03:04.285883 17326 raft_consensus.cc:348] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.286136 17326 raft_consensus.cc:374] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.286228 17326 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.286448 17326 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.287062 17326 ts_tablet_manager.cc:1398] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:04.287341 17326 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:04.288686 17326 tablet_bootstrap.cc:654] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:04.290480 17326 tablet_bootstrap.cc:492] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:04.290599 17326 ts_tablet_manager.cc:1367] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:04.291352 17326 raft_consensus.cc:348] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.291538 17326 raft_consensus.cc:374] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:04.291630 17326 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.291833 17326 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.292367 17326 ts_tablet_manager.cc:1398] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:04.364696 17298 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:04.364903 17298 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.365681 17298 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:04.366173 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:04.366214 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:04.366384 15867 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:04.366469 15999 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:04.366832 15673 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:04.367086 17298 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:04.367198 17298 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:04.367309 17298 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.369297 17298 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.369875 17298 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:04.370353 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:04.370453 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:04.370587 15867 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.370656 15999 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.373440 15867 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:04.373438 15999 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:04.373886 15673 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:04.374217 17298 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:04.374349 17298 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:04.374588 17298 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:04.377725 15587 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:04.508455 17316 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:04.508630 17316 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.509300 17316 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:04.509727 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:04.509727 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:04.509970 15736 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:03:04.509982 15999 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 0.
I20240501 14:03:04.510362 16065 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:04.510625 17316 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:04.510722 17316 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:04.510812 17316 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.512611 17316 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.513221 17316 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:04.513648 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:04.513708 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:04.513885 15736 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.513912 15999 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.516692 15736 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:03:04.516692 15999 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 1.
I20240501 14:03:04.517113 16065 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:04.517403 17316 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:04.517532 17316 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:04.517779 17316 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:04.520677 15587 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 1 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:04.602573 17183 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:04.602792 17183 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.603482 17183 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:04.604080 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:04.604084 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:04.604310 15999 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:04.604346 16128 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:04.604799 15804 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:04.605135 17183 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:04.605226 17183 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:04.605338 17183 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.607546 17183 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.608119 17183 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:04.608503 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:04.608637 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:04.608708 15999 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.608835 16128 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:04.611502 16128 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:04.611502 15999 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:04.611982 15803 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:04.612314 17183 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:04.612689 17183 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:04.612964 17183 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:04.616513 15587 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 1 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:04.789645 17313 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:04.797048 17313 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:04.895048 17316 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:04.902257 17285 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:05.067517 17342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:05.067986 17342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:05.078190 17342 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:03:05.193104 17183 consensus_queue.cc:1035] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:05.212567 17183 consensus_queue.cc:1035] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 264.42 ms
time per row: 0.13221 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:03:06.453568 15590 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:06.463712 15848 tablet_service.cc:1467] Processing CreateTablet for tablet 20717d395e98428594d60999df42298f (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.464215 15848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20717d395e98428594d60999df42298f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.464318 16108 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb17c062d6546f4ab0f7c79cabcaa91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.464473 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb17c062d6546f4ab0f7c79cabcaa91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.464740 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb17c062d6546f4ab0f7c79cabcaa91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.464805 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb17c062d6546f4ab0f7c79cabcaa91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.467684 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 20717d395e98428594d60999df42298f (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.467779 15717 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb17c062d6546f4ab0f7c79cabcaa91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.468123 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20717d395e98428594d60999df42298f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.468828 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb17c062d6546f4ab0f7c79cabcaa91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.468756 16110 tablet_service.cc:1467] Processing CreateTablet for tablet ff5795defe0a42b68392fc5c6a28098a (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.469166 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5795defe0a42b68392fc5c6a28098a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.473484 15979 tablet_service.cc:1467] Processing CreateTablet for tablet 20717d395e98428594d60999df42298f (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.473944 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20717d395e98428594d60999df42298f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.474339 15849 tablet_service.cc:1467] Processing CreateTablet for tablet ff5795defe0a42b68392fc5c6a28098a (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.474736 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5795defe0a42b68392fc5c6a28098a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.479097 17417 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:06.481717 17417 tablet_bootstrap.cc:654] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.483915 15978 tablet_service.cc:1467] Processing CreateTablet for tablet ff5795defe0a42b68392fc5c6a28098a (DEFAULT_TABLE table=rebalance_test_table_8 [id=08de66c0a5d7489f88924b72eee5a691]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:06.484338 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5795defe0a42b68392fc5c6a28098a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:06.490491 17417 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:06.490785 17417 ts_tablet_manager.cc:1367] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 14:03:06.491896 17417 raft_consensus.cc:348] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.492359 17417 raft_consensus.cc:374] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.492543 17417 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.492942 17417 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.493930 17417 ts_tablet_manager.cc:1398] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:03:06.494251 17417 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:06.496572 17417 tablet_bootstrap.cc:654] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.497426 17419 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:06.499821 17419 tablet_bootstrap.cc:654] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.503069 17421 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:06.505362 17417 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:06.505686 17417 ts_tablet_manager.cc:1367] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.006s
I20240501 14:03:06.505702 17421 tablet_bootstrap.cc:654] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.506901 17417 raft_consensus.cc:348] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.507225 17417 raft_consensus.cc:374] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.507301 17418 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:06.507364 17417 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.507755 17417 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.508558 17417 ts_tablet_manager.cc:1398] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:06.508903 17417 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:06.510205 17421 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:06.510298 17419 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:06.510368 17421 ts_tablet_manager.cc:1367] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 14:03:06.510486 17419 ts_tablet_manager.cc:1367] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240501 14:03:06.510804 17417 tablet_bootstrap.cc:654] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.511638 17418 tablet_bootstrap.cc:654] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.511366 17419 raft_consensus.cc:348] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.511720 17419 raft_consensus.cc:374] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.511933 17419 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.512231 17421 raft_consensus.cc:348] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.512238 17419 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.512485 17421 raft_consensus.cc:374] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.512591 17421 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.513123 17419 ts_tablet_manager.cc:1398] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:03:06.513484 17419 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:06.514626 17421 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.515475 17421 ts_tablet_manager.cc:1398] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:03:06.515794 17421 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:06.517040 17419 tablet_bootstrap.cc:654] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.518471 17421 tablet_bootstrap.cc:654] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.524487 17417 tablet_bootstrap.cc:492] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:06.524809 17417 ts_tablet_manager.cc:1367] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20240501 14:03:06.527036 17419 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:06.527391 17419 ts_tablet_manager.cc:1367] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240501 14:03:06.528293 17419 raft_consensus.cc:348] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.528607 17418 tablet_bootstrap.cc:492] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:06.528636 17419 raft_consensus.cc:374] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.528748 17419 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.528868 17418 ts_tablet_manager.cc:1367] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20240501 14:03:06.529085 17419 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.529760 17419 ts_tablet_manager.cc:1398] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:06.529907 17418 raft_consensus.cc:348] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.529947 17417 raft_consensus.cc:348] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.530167 17417 raft_consensus.cc:374] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.530289 17417 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.530297 17418 raft_consensus.cc:374] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.530401 17418 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.530628 17417 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.530717 17418 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.531208 17421 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:06.531337 17418 ts_tablet_manager.cc:1398] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:06.531455 17421 ts_tablet_manager.cc:1367] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240501 14:03:06.531639 17417 ts_tablet_manager.cc:1398] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:03:06.531656 17418 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:06.532239 17421 raft_consensus.cc:348] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.532505 17421 raft_consensus.cc:374] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.532579 17421 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.532861 17421 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.533493 17421 ts_tablet_manager.cc:1398] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:06.533368 17418 tablet_bootstrap.cc:654] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:06.537660 17418 tablet_bootstrap.cc:492] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:06.537843 17418 ts_tablet_manager.cc:1367] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:03:06.538498 17418 raft_consensus.cc:348] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.538699 17418 raft_consensus.cc:374] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:06.538779 17418 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:06.539038 17418 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.539665 17418 ts_tablet_manager.cc:1398] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:06.561534 17411 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:06.561736 17411 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.562433 17411 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:06.562875 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:06.563143 15869 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:06.563087 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:06.563323 16129 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:06.563637 15673 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:06.564033 17411 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:06.564149 17411 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:06.564235 17411 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.566453 17411 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.567242 17411 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:06.567368 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:06.567616 15869 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.567582 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:06.567811 16129 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.569859 16129 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:06.570231 15671 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:06.570297 15869 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:06.570504 17411 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:06.570644 17411 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:06.571074 17411 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:06.574162 15593 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:06.593516 17380 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:06.593730 17380 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.594432 17380 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:06.594974 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:06.594964 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:06.595212 15737 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:06.595245 16000 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 0.
I20240501 14:03:06.595746 15804 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:06.596127 17380 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:06.596261 17380 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:06.596352 17380 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.598276 17380 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.598773 17380 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:06.599269 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:06.599236 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:06.599462 15737 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.599481 16000 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.602341 15737 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:06.602326 16000 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 1.
I20240501 14:03:06.602850 15804 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:06.603246 17380 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:06.603628 17380 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:06.603951 17380 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:06.607050 15593 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 1 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:06.803130 17430 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:06.803335 17430 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.803972 17430 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:06.804517 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:06.804605 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:06.804770 15869 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:06.804828 16129 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:06.805253 15936 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:06.805542 17430 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:06.805678 17430 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:06.805783 17430 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.807734 17430 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.808243 17430 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:06.808660 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:06.808653 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:06.808836 15869 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.808854 16129 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:06.810918 15869 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:06.810951 16129 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:06.811350 15934 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:06.811618 17430 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:06.811754 17430 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:06.812003 17430 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:06.814877 15593 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:06.979686 17410 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:06.984833 17410 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:06.992727 17380 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:06.997330 17380 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:07.215778 17435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:07.216138 17435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:07.223933 17435 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:03:07.368634 17306 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:07.393674 17306 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 261.008 ms
time per row: 0.130504 ms
I20240501 14:03:07.635689 15592 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38782:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:07.646847 15717 tablet_service.cc:1467] Processing CreateTablet for tablet 5160cbc940ba411dbede72e71bc9c851 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.646981 15715 tablet_service.cc:1467] Processing CreateTablet for tablet f6e094d2d5234896b21d838bb8aa4b20 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.647353 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160cbc940ba411dbede72e71bc9c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.647907 15978 tablet_service.cc:1467] Processing CreateTablet for tablet 5160cbc940ba411dbede72e71bc9c851 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.648360 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160cbc940ba411dbede72e71bc9c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.648438 15979 tablet_service.cc:1467] Processing CreateTablet for tablet f6e094d2d5234896b21d838bb8aa4b20 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.648540 15980 tablet_service.cc:1467] Processing CreateTablet for tablet 6c03daaf6d13485c96f88449e2b18fce (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.648847 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e094d2d5234896b21d838bb8aa4b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.648927 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e094d2d5234896b21d838bb8aa4b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.649139 15980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c03daaf6d13485c96f88449e2b18fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.649504 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 5160cbc940ba411dbede72e71bc9c851 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.649897 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160cbc940ba411dbede72e71bc9c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.653165 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 6c03daaf6d13485c96f88449e2b18fce (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.653376 15849 tablet_service.cc:1467] Processing CreateTablet for tablet 6c03daaf6d13485c96f88449e2b18fce (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.653774 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c03daaf6d13485c96f88449e2b18fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.659077 16108 tablet_service.cc:1467] Processing CreateTablet for tablet f6e094d2d5234896b21d838bb8aa4b20 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c70b490b805429d8bf0bef9af0b02e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:07.659492 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e094d2d5234896b21d838bb8aa4b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.663017 17500 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:07.664878 17500 tablet_bootstrap.cc:654] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.665374 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c03daaf6d13485c96f88449e2b18fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.680467 17503 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:07.683368 17502 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Bootstrap starting.
I20240501 14:03:07.684041 17504 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:07.685536 17502 tablet_bootstrap.cc:654] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.685892 17504 tablet_bootstrap.cc:654] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.687736 17503 tablet_bootstrap.cc:654] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.688872 17500 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:07.689093 17500 ts_tablet_manager.cc:1367] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.009s
I20240501 14:03:07.689867 17500 raft_consensus.cc:348] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.690076 17500 raft_consensus.cc:374] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.690156 17500 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.690383 17500 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.690964 17500 ts_tablet_manager.cc:1398] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:07.691555 17500 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:07.692788 17504 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:07.693068 17504 ts_tablet_manager.cc:1367] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 14:03:07.693320 17500 tablet_bootstrap.cc:654] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.694061 17504 raft_consensus.cc:348] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.694365 17504 raft_consensus.cc:374] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.694530 17504 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.694847 17504 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.696350 17504 ts_tablet_manager.cc:1398] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:07.696895 17504 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Bootstrap starting.
I20240501 14:03:07.699348 17504 tablet_bootstrap.cc:654] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.706957 17503 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:07.707182 17503 ts_tablet_manager.cc:1367] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20240501 14:03:07.707912 17504 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: No bootstrap required, opened a new log
I20240501 14:03:07.708146 17504 ts_tablet_manager.cc:1367] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 14:03:07.708106 17503 raft_consensus.cc:348] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.708329 17503 raft_consensus.cc:374] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.708431 17503 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.708700 17503 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.709260 17504 raft_consensus.cc:348] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.709525 17504 raft_consensus.cc:374] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.709636 17504 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.709947 17504 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.710561 17504 ts_tablet_manager.cc:1398] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:07.712828 17503 ts_tablet_manager.cc:1398] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:03:07.715623 17503 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:07.717869 17502 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: No bootstrap required, opened a new log
I20240501 14:03:07.718015 17502 ts_tablet_manager.cc:1367] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.005s
I20240501 14:03:07.718217 17500 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:07.718394 17500 ts_tablet_manager.cc:1367] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.012s
I20240501 14:03:07.718708 17502 raft_consensus.cc:348] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.718865 17502 raft_consensus.cc:374] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.718984 17502 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.719180 17500 raft_consensus.cc:348] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.719198 17502 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.719429 17500 raft_consensus.cc:374] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.719792 17502 ts_tablet_manager.cc:1398] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:07.719991 17503 tablet_bootstrap.cc:654] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.720257 17500 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.720741 17500 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.721346 17500 ts_tablet_manager.cc:1398] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:07.721827 17500 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Bootstrap starting.
I20240501 14:03:07.725796 17500 tablet_bootstrap.cc:654] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.728051 17500 tablet_bootstrap.cc:492] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: No bootstrap required, opened a new log
I20240501 14:03:07.728178 17500 ts_tablet_manager.cc:1367] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 14:03:07.728757 17500 raft_consensus.cc:348] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.728915 17500 raft_consensus.cc:374] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.729035 17500 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.729256 17500 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.729830 17500 ts_tablet_manager.cc:1398] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:07.730887 17503 tablet_bootstrap.cc:492] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:07.731053 17503 ts_tablet_manager.cc:1367] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240501 14:03:07.731679 17503 raft_consensus.cc:348] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.731859 17503 raft_consensus.cc:374] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.731945 17503 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.732825 17503 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.733959 17503 ts_tablet_manager.cc:1398] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:07.734360 17503 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Bootstrap starting.
I20240501 14:03:07.735728 17503 tablet_bootstrap.cc:654] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:07.737428 17503 tablet_bootstrap.cc:492] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: No bootstrap required, opened a new log
I20240501 14:03:07.737552 17503 ts_tablet_manager.cc:1367] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:03:07.738135 17503 raft_consensus.cc:348] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.738312 17503 raft_consensus.cc:374] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:07.738415 17503 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Initialized, Role: FOLLOWER
I20240501 14:03:07.738608 17503 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.739127 17503 ts_tablet_manager.cc:1398] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:03:07.793943 17430 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:07.794188 17430 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.794771 17430 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:07.795266 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:07.795359 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" is_pre_election: true
I20240501 14:03:07.795508 15737 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:07.795595 15867 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:07.796034 15935 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:07.796378 17306 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:07.796491 17306 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:07.796598 17306 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.798622 17306 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.799193 17306 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:07.799626 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:07.799652 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:07.799806 15737 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.799851 15867 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.802115 15867 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:07.802219 15737 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:07.802459 15936 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:07.802721 17306 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:07.802873 17306 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:07.803117 17306 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:07.806212 15592 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:07.835880 17411 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:07.836120 17411 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.836875 17411 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:07.837628 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:07.837821 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:07.837901 16130 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:07.838089 15999 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 0.
I20240501 14:03:07.838627 15672 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:07.839076 17411 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:07.839179 17411 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:07.839196 17306 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:07.839298 17411 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.839371 17306 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.839936 17306 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:07.840363 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:07.840598 16130 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:07.840763 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205" is_pre_election: true
I20240501 14:03:07.840997 15737 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 0.
I20240501 14:03:07.841428 15935 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:07.841993 17306 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:07.842018 17411 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.842110 17306 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:07.842197 17306 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.842547 17411 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:07.842967 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:07.843051 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:07.843163 16130 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.843283 15999 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.845260 17306 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.845868 16130 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:07.846372 15999 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 1.
I20240501 14:03:07.846479 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:07.846653 16130 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.846871 15672 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:07.847198 17411 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:07.847716 17411 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:07.847957 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:07.848031 17411 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:07.848228 15737 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:07.849640 16130 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:07.850106 15934 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:07.850381 17430 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:07.850805 17430 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:07.851128 17430 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:07.851948 15592 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 1 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:07.854439 15589 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 1 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:07.855160 15737 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 1.
I20240501 14:03:07.855459 17306 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 1 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:08.228139 17411 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:08.234025 17411 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:08.270478 17517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:08.270809 17517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:03:08.273542 17306 consensus_queue.cc:1035] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:08.280184 17515 consensus_queue.cc:1035] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:08.283087 17517 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:03:08.387145 17515 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:08.393697 17515 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 273.421 ms
time per row: 0.13671 ms
I20240501 14:03:08.737501 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.69:37923
--local_ip_for_outbound_sockets=127.22.33.69
--tserver_master_addrs=127.22.33.126:42697
--webserver_port=44523
--webserver_interface=127.22.33.69
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:03:09.097054 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:09.097491 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:09.097656 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:09.105580 17580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:03:09.105706 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:09.105866 17580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.69
I20240501 14:03:09.115470 17580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.69:37923
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.33.69
--webserver_port=44523
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.69
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:09.116942 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:09.119284 17580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:09.132428 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:09.132831 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:09.133585 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:09.134275 17580 server_base.cc:1034] running on GCE node
I20240501 14:03:09.134999 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:09.136186 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:09.137420 17580 hybrid_clock.cc:648] HybridClock initialized: now 1714572189137356 us; error 43 us; skew 500 ppm
I20240501 14:03:09.137826 17580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:09.140806 17580 webserver.cc:460] Webserver started at http://127.22.33.69:44523/ using document root <none> and password file <none>
I20240501 14:03:09.141806 17580 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:09.141963 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:09.147773 17580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 14:03:09.150926 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.152405 17580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:09.152570 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "acf31b7bd4974dc4951935d1c2da8f26"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:03:09.153514 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:09.181658 17580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:09.182588 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:09.182873 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:09.183828 17580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:09.185359 17580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:09.185437 17580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.185568 17580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:09.185623 17580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.220659 17580 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.69:37923
I20240501 14:03:09.220691 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.69:37923 every 8 connection(s)
I20240501 14:03:09.222358 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:03:09.231240 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 17580
I20240501 14:03:09.231885 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.70:44387
--local_ip_for_outbound_sockets=127.22.33.70
--tserver_master_addrs=127.22.33.126:42697
--webserver_port=43099
--webserver_interface=127.22.33.70
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:03:09.236029 17708 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:03:09.236336 17708 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:09.237149 17708 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:03:09.238762 15593 ts_manager.cc:194] Re-registered known tserver with Master: acf31b7bd4974dc4951935d1c2da8f26 (127.22.33.69:37923)
I20240501 14:03:09.239662 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.69:40231
W20240501 14:03:09.599323 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:09.599607 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:09.599743 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:09.608311 17713 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:03:09.608441 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:09.608606 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.70
I20240501 14:03:09.619297 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.70:44387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.33.70
--webserver_port=43099
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.70
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:09.620765 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:09.622565 17713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:09.634846 17719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:09.635480 17722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:09.636015 17720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:09.636749 17713 server_base.cc:1034] running on GCE node
I20240501 14:03:09.637434 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:09.638314 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:09.639503 17713 hybrid_clock.cc:648] HybridClock initialized: now 1714572189639452 us; error 48 us; skew 500 ppm
I20240501 14:03:09.639794 17713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:09.641988 17713 webserver.cc:460] Webserver started at http://127.22.33.70:43099/ using document root <none> and password file <none>
I20240501 14:03:09.642761 17713 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:09.642976 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:09.649917 17713 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:03:09.653332 17729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.655036 17713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:03:09.655220 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bc2fcc831acb4607b85e5e6c328182e4"
format_stamp: "Formatted at 2024-05-01 14:02:50 on dist-test-slave-3610"
I20240501 14:03:09.656479 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:09.672120 17713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:09.672899 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:09.673189 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:09.674103 17713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:09.675577 17713 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:09.675648 17713 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.675774 17713 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:09.675822 17713 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:09.711542 17713 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.70:44387
I20240501 14:03:09.711576 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.70:44387 every 8 connection(s)
I20240501 14:03:09.712872 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:03:09.717234 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 17713
I20240501 14:03:09.717792 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.71:39595
--local_ip_for_outbound_sockets=127.22.33.71
--tserver_master_addrs=127.22.33.126:42697
--webserver_port=33645
--webserver_interface=127.22.33.71
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:03:09.726670 17842 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:03:09.727052 17842 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:09.727968 17842 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:03:09.729661 15593 ts_manager.cc:194] Re-registered known tserver with Master: bc2fcc831acb4607b85e5e6c328182e4 (127.22.33.70:44387)
I20240501 14:03:09.731098 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.70:40955
W20240501 14:03:10.094877 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:10.095171 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:10.095296 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:10.103955 17845 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:03:10.104089 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:10.104264 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.71
I20240501 14:03:10.114624 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.71:39595
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.33.71
--webserver_port=33645
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.71
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:10.116338 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:10.118531 17845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:10.137372 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:10.138419 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:10.142904 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:10.143576 17845 server_base.cc:1034] running on GCE node
I20240501 14:03:10.144229 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:10.145381 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:10.146600 17845 hybrid_clock.cc:648] HybridClock initialized: now 1714572190146509 us; error 88 us; skew 500 ppm
I20240501 14:03:10.146963 17845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:10.154871 17845 webserver.cc:460] Webserver started at http://127.22.33.71:33645/ using document root <none> and password file <none>
I20240501 14:03:10.155716 17845 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:10.155872 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:10.162859 17845 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 14:03:10.166426 17861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.168123 17845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240501 14:03:10.168287 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c747f91e33d9460f92d1fd00f12b4bba"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:03:10.169301 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:10.188436 17845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:10.189363 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:10.189628 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:10.190618 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:10.192240 17845 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:10.192339 17845 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.192472 17845 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:10.192528 17845 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.239423 17845 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.71:39595
I20240501 14:03:10.239476 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.71:39595 every 8 connection(s)
I20240501 14:03:10.240913 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:03:10.242468 17708 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
I20240501 14:03:10.243131 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 17845
I20240501 14:03:10.243831 22661 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF0nCis/build/asan/bin/kudu
/tmp/dist-test-taskF0nCis/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.33.72:35717
--local_ip_for_outbound_sockets=127.22.33.72
--tserver_master_addrs=127.22.33.126:42697
--webserver_port=39113
--webserver_interface=127.22.33.72
--builtin_ntp_servers=127.22.33.84:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:03:10.266602 17975 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:03:10.266937 17975 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:10.267750 17975 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:03:10.269294 15593 ts_manager.cc:194] Re-registered known tserver with Master: c747f91e33d9460f92d1fd00f12b4bba (127.22.33.71:39595)
I20240501 14:03:10.270179 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.71:50509
W20240501 14:03:10.603262 17979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:10.603591 17979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:10.603737 17979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:10.611459 17979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:03:10.611619 17979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:10.611789 17979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.33.72
I20240501 14:03:10.622014 17979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.33.84:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.33.72:35717
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.33.72
--webserver_port=39113
--tserver_master_addrs=127.22.33.126:42697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.33.72
--log_dir=/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:10.623425 17979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:10.625262 17979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:10.637593 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:10.638052 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:10.638731 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:10.639573 17979 server_base.cc:1034] running on GCE node
I20240501 14:03:10.640266 17979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:10.641489 17979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:10.642743 17979 hybrid_clock.cc:648] HybridClock initialized: now 1714572190642686 us; error 46 us; skew 500 ppm
I20240501 14:03:10.643100 17979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:10.650804 17979 webserver.cc:460] Webserver started at http://127.22.33.72:39113/ using document root <none> and password file <none>
I20240501 14:03:10.651727 17979 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:10.651859 17979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:10.657783 17979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 14:03:10.660809 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.662705 17979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:03:10.662946 17979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "340d8cf614aa438c94feebe2ddddc00f"
format_stamp: "Formatted at 2024-05-01 14:02:51 on dist-test-slave-3610"
I20240501 14:03:10.664117 17979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:10.707741 17979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:10.708582 17979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:10.708851 17979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:10.709825 17979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:10.711664 17979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:10.711740 17979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.711840 17979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:10.711889 17979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:10.736037 17842 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
I20240501 14:03:10.747944 17979 rpc_server.cc:307] RPC server started. Bound to: 127.22.33.72:35717
I20240501 14:03:10.747988 18109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.33.72:35717 every 8 connection(s)
I20240501 14:03:10.749521 17979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF0nCis/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714571948533166-22661-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:03:10.752857 22661 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF0nCis/build/asan/bin/kudu as pid 17979
I20240501 14:03:10.771334 18110 heartbeater.cc:347] Connected to a master server at 127.22.33.126:42697
I20240501 14:03:10.771677 18110 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:10.772593 18110 heartbeater.cc:511] Master 127.22.33.126:42697 requested a full tablet report, sending...
I20240501 14:03:10.774112 15593 ts_manager.cc:194] Re-registered known tserver with Master: 340d8cf614aa438c94feebe2ddddc00f (127.22.33.72:35717)
I20240501 14:03:10.775111 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.33.72:50625
I20240501 14:03:10.803270 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.803576 15734 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.803712 15734 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.805662 15734 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.806353 15734 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.806772 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.806931 15869 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.808879 15869 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.809439 15673 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:10.809715 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.809861 17563 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.809957 16000 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.809998 17563 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.810047 16000 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.810288 17563 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.810360 16000 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.810904 16000 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.813490 16000 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.813730 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.813995 15734 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.814157 15734 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.814975 15593 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:10.817087 15734 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.818409 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.818559 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.818677 16000 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.818802 15869 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.818925 15869 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:10.819291 15869 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.819986 15869 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.822463 16000 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.823354 15672 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:10.823778 17982 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.823951 17982 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.824308 17982 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.825060 15869 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.825639 15734 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.827014 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.827277 15734 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.827430 15734 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.831352 15734 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.833602 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.833869 16000 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.833959 15734 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:10.837106 16000 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.837105 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.837324 16128 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.837440 16128 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:10.837672 15672 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:10.837806 16128 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.837992 17982 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.838166 17982 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.838359 16128 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.838418 17982 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.840611 15593 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:10.841696 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.842461 16128 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.843101 15734 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.843231 15734 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.847450 15593 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:10.846803 15734 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.850142 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.850168 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.850409 16128 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.850529 16000 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.850709 16000 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.851184 16000 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.851632 16000 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.851634 15734 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:10.852677 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.852864 15734 raft_consensus.cc:465] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.853096 16128 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.853547 15671 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.853832 17982 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.854053 17982 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.854388 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.854578 15734 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.854700 15734 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.854411 17982 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.855304 16000 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.857931 15734 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.858170 15593 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:10.859053 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.859278 15869 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.861289 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.861470 16128 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.861572 16128 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:10.861837 16128 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.862154 16128 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.864120 16128 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.864560 15671 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.865054 15869 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.865097 17982 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.865154 15734 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:10.865485 17982 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.865808 17982 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.868901 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.869812 15737 raft_consensus.cc:465] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.869843 15593 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:10.870713 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.870887 15737 raft_consensus.cc:465] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.871984 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.872182 15737 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.872278 15737 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.874943 15737 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.876117 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.876317 16128 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.876259 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.876415 16128 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:10.876451 16000 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.876672 16128 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.876024 15737 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.877013 16128 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.879026 16128 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.879307 16000 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.879462 15671 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.880046 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.880123 17982 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.880278 17982 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.880362 15737 raft_consensus.cc:465] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.880641 17982 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.881287 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.881449 15737 raft_consensus.cc:465] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.882275 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.882434 15737 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.882588 15737 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.885320 15737 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.885977 15737 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.886339 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.886535 16128 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.886675 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.886921 16000 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.887019 16000 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.887099 15593 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:10.887351 16000 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.887751 16000 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.890725 16000 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.891191 15672 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:10.891281 16128 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.891772 17982 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.891952 17982 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.892349 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.892553 15737 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.892668 15737 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.893020 17982 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.895579 15737 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.896210 15593 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:10.896960 15737 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:10.897115 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.897347 15869 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.897501 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.897717 16000 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.897804 16000 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.898074 16000 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.898428 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.899510 15869 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.899977 15673 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:10.900962 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.901227 15734 raft_consensus.cc:465] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.901293 16000 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.901494 17982 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.901691 17982 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.901973 17982 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:10.902850 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.903054 15734 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.903213 15734 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.905206 15734 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.906296 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.906522 16128 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.906625 16128 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:10.906793 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.906991 15869 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.906909 16128 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.907312 16128 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.909799 15869 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.910037 16128 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.910269 15673 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:10.910567 17982 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.910774 17982 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.911069 17982 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.910809 15734 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:10.912868 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.913128 15737 raft_consensus.cc:465] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.914219 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.914364 15737 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.914463 15737 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.916225 15737 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.916854 15737 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:10.916945 15593 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:10.917644 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.917877 15867 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.917971 15867 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:10.917995 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.918224 16128 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.918227 15867 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.918587 15867 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.920850 16128 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.921221 15867 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.921517 15671 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.921517 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.921917 15737 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.922087 15737 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.922118 17982 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.922283 17982 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.922609 17982 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:10.924336 15593 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:10.925384 15737 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.925998 15737 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.926896 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.927148 16000 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.927238 16000 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.927510 16000 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.927778 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.927886 16000 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.927834 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.928025 16128 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.928002 15737 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.928185 15737 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.930912 16128 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.930912 15737 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.931339 16000 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.931355 15671 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.931680 17981 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.931730 15737 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:10.931908 17981 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.932281 17981 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.932864 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.933143 15734 raft_consensus.cc:465] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.934217 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.934613 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.934688 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:10.934861 15999 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.934880 15867 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.934984 15999 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.935297 15999 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.935765 15999 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.938608 15867 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.939092 15999 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.939097 15673 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:10.939450 15734 raft_consensus.cc:465] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.939718 17982 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.939962 17982 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.940255 17982 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:10.940548 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.940749 15734 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.940909 15734 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.944550 15734 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.945673 15734 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:10.946063 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:10.946319 16128 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.946754 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:10.946962 15736 raft_consensus.cc:465] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:10.948673 16128 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.949141 15671 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:10.949412 17563 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:10.950831 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:10.951109 15999 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:10.951316 15999 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:10.951694 15999 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.952148 15999 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:10.955123 15999 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 2.
I20240501 14:03:10.961565 17563 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:10.961894 17563 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:10.967520 15593 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:10.968017 15593 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:10.968467 15593 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:10.975179 15593 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 2 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:10.996891 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:10.997227 15999 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:10.997395 15999 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.000293 15999 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.001121 15999 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.001754 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.001983 16128 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.002043 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.002295 15736 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.002386 15736 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.002477 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.002740 15999 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.002835 15999 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.002707 15736 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.003158 15736 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.004875 16128 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.005702 15934 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.005642 15999 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.006029 17846 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.006183 17846 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.006263 15736 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.006487 17846 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.006764 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.006939 15736 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.007020 15736 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.007144 15999 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.007637 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.007853 15867 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.009167 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.009479 15999 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.009693 15999 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.010712 15867 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.011250 15936 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.011638 17430 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.013659 15999 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.014321 15736 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.015012 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.015282 15867 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.015810 15736 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.019202 15867 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.019773 15936 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.021304 15736 raft_consensus.cc:2396] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.025540 15999 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.025740 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.025979 15736 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.026118 15736 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.026289 15593 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.026561 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.026757 16000 raft_consensus.cc:465] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:11.026460 15736 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.027266 15736 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.028270 17846 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.028452 17846 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.028713 17846 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.029891 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.030200 16000 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.030344 16000 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.032476 15736 raft_consensus.cc:2396] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.034197 17430 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.034521 17430 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.041527 16000 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.042925 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.042701 16000 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.043159 16128 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.044196 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.044376 16000 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.044487 16000 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.046022 16128 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.046407 15934 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.046509 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.046736 15869 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.046816 15869 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.047099 15869 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.047477 15869 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.049554 17430 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.050128 15869 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.049700 17430 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.051323 17430 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.054059 16000 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.055552 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.055802 15736 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.055922 15736 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.058864 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.057215 15736 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.059057 16128 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.059182 15736 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.060513 16000 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.061812 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.062013 16128 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.062043 15999 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.062170 15999 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.062630 15934 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.062920 18142 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.063925 18142 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.064363 18142 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.066861 15999 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.068147 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.068401 15737 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.068495 15737 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.068785 15737 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.069205 15737 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.072014 15737 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.072541 15935 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:11.073647 15999 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.074232 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.074601 15867 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.074611 17430 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.074707 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.074913 15999 raft_consensus.cc:465] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:11.075709 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.075930 16000 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.076049 16000 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.077883 15867 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.078672 16000 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.074916 17430 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.079649 17430 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.079802 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.080015 16128 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.080101 16128 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.080381 16128 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.080785 16128 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.083920 16128 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.084017 15736 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.084770 15934 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.085533 17846 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.085812 17846 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.086089 15593 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.086206 17846 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.086720 15593 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.095427 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.095672 15867 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.098966 15867 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.101613 16000 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.104794 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.105217 16000 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.105412 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.106911 15593 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.107497 15593 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.107793 16000 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.107975 15593 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.108513 15593 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.109104 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.109190 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.109349 15736 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.109447 15736 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.109591 15867 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.109726 15736 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.110113 15736 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.112565 15867 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.112946 15736 raft_consensus.cc:2396] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.113142 15936 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.113667 17846 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.113842 17846 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.114089 17846 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.115337 16000 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.117017 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.117228 16000 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.117352 16000 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.119386 16000 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.119633 15593 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.120138 16000 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.125583 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.125798 16128 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.127733 16128 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.128197 15934 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.128361 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.128593 15736 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.128702 15736 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.128780 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.128947 16000 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.129096 16000 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.128998 15736 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.129344 15736 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.131898 16000 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.132128 15736 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.132450 17846 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.132596 17846 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.132858 17846 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.133145 16000 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.133625 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.133812 16128 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.134482 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.134748 16000 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.134925 16000 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.136847 16128 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.137264 15934 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.137625 18141 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.137773 18141 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.138064 18141 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.138409 16000 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.139479 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.139720 15736 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.139851 15736 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.139783 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.140003 15737 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.140080 15737 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.140151 16000 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.140450 15737 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.140751 15737 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.140744 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.141000 16128 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.141738 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.141991 16000 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.142168 16000 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.143644 15737 raft_consensus.cc:2396] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.143931 16128 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.144654 15934 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.145416 17430 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.145634 17430 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.145685 16000 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.145938 17430 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.146402 16000 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.146870 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.146888 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.147076 16128 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.147084 15737 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.147161 15737 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.147454 15737 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.147763 15737 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.147739 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.148042 16000 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.148218 16000 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.150117 16128 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.150540 15737 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.150585 15934 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.150892 18141 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.151062 18141 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.151345 18141 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.151683 16000 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.152379 16000 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.152875 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.152904 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.153115 16130 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.153153 15867 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.153262 15867 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.153549 15867 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.153950 15867 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.156906 15867 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.156664 16130 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.157795 15936 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.158483 15736 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.158931 18142 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.160473 15736 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.166532 15736 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.176314 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.176589 16000 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.176751 16000 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.179961 16000 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.180763 16000 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.181484 15593 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.181942 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.182138 15999 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.182276 15999 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.185791 15999 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.181366 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.186317 15737 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.186487 15737 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.187026 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.186962 15737 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.187296 16128 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.187637 15737 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.190448 16128 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.191040 15934 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.191401 17846 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.192310 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.192466 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.192579 15734 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.192662 15867 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.192668 15734 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.193089 15734 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.193485 15734 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.195492 15867 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.197297 15734 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.191561 15999 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.196134 15936 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.196650 18142 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.198014 17846 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.198261 17846 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.198257 18142 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.198617 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.198853 15999 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.198972 15999 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.199487 18155 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.199662 18155 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.200070 18155 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.202033 15999 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.202466 15737 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.204130 15999 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.204721 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.205026 15867 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.205377 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.205720 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.206030 16000 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.206367 15736 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.206183 16000 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.207330 15736 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.207813 15867 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.208326 15936 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.207751 15736 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.208839 15736 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.208942 15593 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.209668 15593 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.210204 15593 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.210757 15593 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.212504 18155 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.213232 18155 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.213518 18155 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.213866 16000 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.215147 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.216022 15737 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.216150 15737 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.216511 15737 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.216797 15737 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.219724 15737 raft_consensus.cc:2396] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.220641 15736 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.221433 16000 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.222069 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.222275 15867 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.222476 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.222684 15999 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.222805 15999 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.225114 15867 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 3.
I20240501 14:03:11.225968 15936 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.225944 15999 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.226487 18142 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.226645 18142 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.226917 18142 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.227121 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.227350 16128 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.230197 16128 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.230760 15934 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.231079 17846 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.231237 17846 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.231524 17846 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.233294 15999 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.233413 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.233668 15734 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.233745 15734 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.234035 15734 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.234329 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.234532 15734 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.234594 15998 raft_consensus.cc:465] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:11.235603 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.235821 15998 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.235970 15998 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.239526 15998 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.239679 15734 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.242167 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.242503 15867 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.242597 15998 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.243857 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:11.244096 15999 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.244217 15999 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.244875 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.245126 15734 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.245220 15734 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.245474 15734 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.245932 15734 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.249029 15867 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.248943 15999 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.249596 15936 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.250468 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.250530 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.250733 15869 raft_consensus.cc:3032] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.250820 15869 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.250873 16128 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.251176 15869 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.251645 15869 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.254570 16128 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.255087 15934 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.255250 15869 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.255503 18154 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.255667 18154 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.255774 17430 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.256001 17430 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.256525 17430 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.256032 18154 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.259199 15999 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.261713 15734 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 2.
I20240501 14:03:11.267449 15593 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.268069 15593 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.268450 15593 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.273221 17975 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
I20240501 14:03:11.277582 15593 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 3 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.278005 15593 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.278404 15593 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.278774 15593 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 2 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.286442 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.286737 16128 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.286870 16128 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.289419 16128 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.290508 16128 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.290961 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.291201 15999 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.291311 15999 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.291603 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.291774 16128 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.291638 15999 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.291851 16128 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.292120 15999 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.294625 16128 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.295899 15999 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.295929 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.296200 15734 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.296631 16065 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.296834 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.296967 17483 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.297168 15869 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.299549 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.299759 15999 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.299860 15999 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.300175 15734 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.300171 15999 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.300508 15999 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.301064 17483 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.300916 15869 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.301380 17483 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.304782 15999 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.305061 16128 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.305330 16065 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.305851 17483 raft_consensus.cc:2726] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.305650 15593 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.306152 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.306354 16130 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.306478 16130 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.308604 16130 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.309317 16130 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.309746 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.309999 15869 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.311734 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.311942 16130 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.312062 16130 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.312721 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.312930 15999 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.313124 15999 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.313429 15999 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.313683 15869 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.313753 15999 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.314952 16130 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.315753 16130 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.316155 16066 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.316465 18169 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.316427 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.316624 15734 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.316723 15734 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.317019 15734 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.317413 15734 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.318446 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.318617 16130 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.318718 16130 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.319525 15999 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.320827 15734 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.321043 16130 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.321321 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.321573 15867 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.321610 16130 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.321831 18169 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.322144 18169 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.324368 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.324599 16130 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.324764 16130 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.324831 15867 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.325430 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.325647 15869 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.325443 15593 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.326483 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.326740 15999 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.326833 15999 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.326886 16065 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.327076 15999 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.327431 15999 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.328162 15869 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.328586 16066 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.328915 18175 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.329209 17483 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.329874 15869 raft_consensus.cc:1261] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.330350 15999 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.330941 17982 consensus_queue.cc:1035] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.331746 16130 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.331970 18175 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.332502 16130 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.332468 18175 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.334721 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.334951 15734 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.337330 15734 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.337728 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.337961 15999 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.338114 15999 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.338377 17483 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.338407 15999 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.338721 15999 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.338764 17483 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.340653 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.340868 16130 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.341053 16130 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.342680 15593 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.343487 15593 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.344039 16130 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.344758 16130 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.347218 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.347463 15867 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.348177 15999 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.348063 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.348341 15734 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.348459 15734 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.348826 15734 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.349339 15734 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.350276 15867 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.350800 16066 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.351117 17483 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.351397 17483 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.351802 16065 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.351711 17483 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.352130 18180 raft_consensus.cc:2726] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.352557 16128 raft_consensus.cc:1261] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.353564 15734 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.354164 17563 consensus_queue.cc:1035] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.356575 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.356778 16130 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.356886 16130 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.358971 16130 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.359565 16130 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.359897 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.360404 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.360776 15593 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.364593 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.364828 16130 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.365008 16130 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.372941 16129 raft_consensus.cc:1226] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 2. Current term is 3. Ops: []
I20240501 14:03:11.373123 15734 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.373235 15734 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.373926 16130 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.373991 15734 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.374378 15734 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.374686 16130 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.375401 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.375645 15737 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
W20240501 14:03:11.374838 17563 consensus_queue.cc:1159] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" responder_term: 3 status { last_received { term: 1 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 14:03:11.379001 15737 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.379937 15734 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.380537 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.380812 16130 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.380838 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.381098 15999 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.381055 16065 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:11.381057 16130 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.381203 15999 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.381476 15999 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.381600 17483 raft_consensus.cc:2726] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 14:03:11.381812 15999 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.382119 15869 raft_consensus.cc:1261] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.382719 17982 consensus_queue.cc:1035] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.384765 15999 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.384685 16130 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.385336 16065 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.385797 17483 raft_consensus.cc:2726] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.386879 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.387248 15869 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.388923 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.389209 15999 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.389427 15999 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.389761 15999 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.390208 15999 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.393810 15999 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.394492 16130 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.395689 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.395967 16130 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.396108 16130 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.397119 15869 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.397575 16066 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:11.397986 17483 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.398154 17483 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.398470 17483 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.398619 16130 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.399276 16130 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.400312 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.400532 15999 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.400652 15999 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.400569 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.401050 15734 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.401055 15999 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.401464 15999 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.404918 15999 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.405193 15734 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.405787 16065 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.405925 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.406193 16130 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.406302 18180 raft_consensus.cc:2726] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.406334 16130 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.408834 16130 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.408905 15593 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.409600 16130 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.410322 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.410559 15869 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.411371 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.411552 16128 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.411659 16128 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.412787 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.413000 15734 raft_consensus.cc:3032] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.413133 15734 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.413450 15734 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.413833 15734 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.413918 16128 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.414495 16128 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.415458 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.415414 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.415690 16130 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.415850 16130 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.416051 15999 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.416169 15999 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.416193 15869 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.416716 16066 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.416877 15999 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.417147 17483 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.417225 15999 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.417716 17483 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.418020 17483 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.418697 16130 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.419349 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.419571 15737 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.420079 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.420248 16000 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.420359 16000 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.420583 16000 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.420845 16000 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.422618 15737 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.422981 16065 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.423312 18180 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.423379 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.423560 18180 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.423620 15736 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.423914 18180 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.426160 16130 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.427177 15736 raft_consensus.cc:2396] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.427529 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.427811 16130 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.427958 16130 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.428402 16000 raft_consensus.cc:2396] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.429738 16065 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.430250 17483 raft_consensus.cc:2726] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.430649 15734 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.430465 15593 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.431049 15999 raft_consensus.cc:2396] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.431463 16130 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.432190 16130 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.432704 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.432955 15734 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.433706 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.434285 16130 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.434451 16130 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.435592 15736 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.435842 15736 pending_rounds.cc:76] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:11.436026 15736 pending_rounds.cc:97] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.436273 15736 raft_consensus.cc:2866] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.436605 15734 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.437465 16130 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.437661 18155 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.438683 16130 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.438853 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.439100 15869 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.439204 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.439463 16000 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.439584 16000 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.439805 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.440017 16130 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.439941 16000 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.440119 16130 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.440316 16000 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.442129 15869 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.442651 16066 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.442938 17483 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.443245 17483 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.443539 17483 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.443773 16130 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.444221 16000 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.444717 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.444958 15999 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.445099 15999 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.445478 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.446022 15736 raft_consensus.cc:3032] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.446628 15999 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.447029 15999 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.446197 15736 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.447630 15736 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.448298 15736 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.452534 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.453087 15869 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.457123 16130 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.457918 15869 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.458580 15736 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.459321 16065 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.459774 18180 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.459985 18180 raft_consensus.cc:686] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.460422 18180 consensus_queue.cc:237] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.463856 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.464129 16130 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.464273 16130 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.465070 15593 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.465834 15593 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.467031 15999 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.467235 15869 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.467506 16065 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:11.467872 18180 raft_consensus.cc:2726] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.468428 18155 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.471750 16130 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.472251 15593 catalog_manager.cc:5484] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.473711 16130 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.474341 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.474377 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.474529 15869 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.474718 15734 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.474885 15734 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.475342 15734 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.475703 15734 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.476668 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.476853 16128 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.476997 16128 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.477248 15869 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.477780 16066 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.478082 18180 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.478240 18180 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.478528 18180 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.479324 15734 raft_consensus.cc:2396] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.479823 16128 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.484763 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.484822 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.485014 15869 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.485074 15734 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.485174 15734 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:11.485455 15734 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.485826 15734 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.486140 15867 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.486984 17846 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.487959 15869 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.488389 16066 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.488761 17483 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.488759 15734 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.488909 17483 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.489261 17483 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.490161 16128 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.492806 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.493079 16130 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.493248 16130 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.494654 15734 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.495321 18142 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.496171 16130 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.497119 16130 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.498507 15867 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.499207 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.499450 15737 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.499606 18142 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.499766 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.499948 16130 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.500090 16130 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.502393 15737 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.502879 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.502920 16130 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.503105 15999 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.503218 15999 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.503511 15999 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.503885 15999 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.504069 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.504316 15869 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.505465 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.505653 16000 raft_consensus.cc:3032] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:11.505749 16000 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.506019 16000 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.506408 16000 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:11.507339 15869 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.507788 16066 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.508119 17483 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:11.508363 17483 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.508666 17483 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.509786 16065 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.510147 18175 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.510373 18175 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.510692 18175 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.512836 15999 raft_consensus.cc:2396] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.513619 16000 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 2.
I20240501 14:03:11.514223 16130 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 2 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.515503 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:11.515728 16130 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.515857 16130 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.518574 16130 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.519708 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:11.519968 15869 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.524583 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.524976 15999 raft_consensus.cc:3032] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.525115 15999 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.525418 15999 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.525771 15999 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.526465 15869 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 3.
I20240501 14:03:11.527566 16066 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.527959 18169 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.528527 18169 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.528653 16130 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:11.528885 18169 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.532145 15999 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.540890 15736 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.541115 15736 pending_rounds.cc:76] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:11.541219 15736 pending_rounds.cc:97] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.541471 15736 raft_consensus.cc:2866] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.542171 18155 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.551980 15867 raft_consensus.cc:1261] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.552664 18155 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.561511 15591 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.570542 15591 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.570996 15591 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.571342 15591 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 3 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.571650 15591 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 1 to 2, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 2 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.574493 15736 raft_consensus.cc:1261] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.574617 15736 pending_rounds.cc:76] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:11.574673 15736 pending_rounds.cc:97] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.574750 15736 raft_consensus.cc:2866] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.575742 17846 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.609990 15867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.610311 15867 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.610441 15867 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.614151 15867 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.615620 15867 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.616619 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.616885 15736 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.619721 15736 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.620159 15804 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.620483 17980 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.620599 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.620826 15999 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.620935 15999 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.624248 15999 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.624642 15999 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.627574 15999 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.628719 15867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.628950 15867 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.629099 15867 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.631929 15867 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.633173 15869 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.634068 17846 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.635387 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.635607 15999 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.638907 15999 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.639348 15804 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.641534 18226 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.641710 18226 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.642077 18226 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.646840 15736 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.647534 18155 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.647987 15867 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.648150 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.648365 16130 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.648491 16130 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.648763 16130 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.648917 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.649117 15869 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.649227 15869 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.649516 16130 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.652119 15869 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.653324 16130 raft_consensus.cc:2396] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.653236 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.653472 15737 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.656419 15737 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.657413 15804 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.657704 18226 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.657888 18226 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.658181 18226 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.658527 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.658757 15999 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.658852 15999 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.659164 15999 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.659464 15868 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.659524 15999 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.660097 18155 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.662649 15869 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.665108 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.665975 15999 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.667271 15866 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.667414 15866 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.668395 15736 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.669129 18155 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.670215 15866 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.671234 15866 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.671348 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.671550 16130 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.671646 16130 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.671945 16130 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.672295 16130 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.672523 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.672719 15866 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.672814 15866 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.675405 16130 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.675578 15866 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.676841 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.677126 16130 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.677194 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.677292 16130 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.677418 15737 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.677737 16130 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.678102 16130 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.680467 15737 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.680960 15804 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.681422 18228 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.681557 18228 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.681855 18228 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.682224 16130 raft_consensus.cc:2396] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.685523 15868 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.687225 17846 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.688247 15737 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.688783 17846 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.698190 15866 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.699409 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.699761 15869 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.699980 15869 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.704181 15869 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.705212 15869 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.705626 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.705843 16130 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.705971 16130 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.706010 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.706209 16000 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.706245 16130 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.706569 16130 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.709403 16130 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.710021 16000 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.710505 15804 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.710795 18226 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.710929 18226 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.711149 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.711350 15869 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.711230 18226 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.711462 15869 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.714330 15867 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.715004 18142 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.720201 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.720482 15999 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.728230 15999 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.728855 15804 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.737989 15867 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.740134 18180 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.740993 18241 raft_consensus.cc:2726] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.741618 15869 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.742303 15869 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.747822 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.748098 15737 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.750880 15737 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.752390 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.752627 16000 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.752710 16000 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.753057 16000 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.753417 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.756094 16000 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.759228 15737 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.759353 15737 pending_rounds.cc:76] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:11.759431 15737 pending_rounds.cc:97] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.759507 15737 raft_consensus.cc:2866] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.759639 15804 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.760273 18226 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.760164 18142 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.760468 18226 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.760728 18226 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.761176 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.761420 15869 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.761547 15869 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.766440 17980 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.766856 17980 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.766223 15869 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.769865 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.770100 16130 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.770216 16130 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.770606 15869 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.770503 16130 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:11.770705 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.770830 16130 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.770962 15999 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.771704 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.771927 15866 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.772109 15866 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.773998 16130 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.775743 15866 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.775938 15999 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.776783 15866 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.774603 15593 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.778132 18110 heartbeater.cc:503] Master 127.22.33.126:42697 was elected leader, sending a full tablet report...
I20240501 14:03:11.780133 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.780350 15999 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.780434 15999 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.780709 15999 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.781208 15999 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.782797 15804 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.783156 17980 raft_consensus.cc:2726] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:11.783605 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.783840 15869 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.784037 15869 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.787647 15869 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.789135 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.789224 15869 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.789441 16130 raft_consensus.cc:3032] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.789559 16130 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.789876 16130 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.790174 16130 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.790477 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.791644 15999 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.792116 15804 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.792318 16130 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.792492 18226 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.793100 15869 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.793277 15869 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.796790 15593 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.797398 15593 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.797708 15737 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.797857 15593 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.798287 15593 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.798753 17483 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.798754 15593 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:11.802901 15869 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.803761 15866 raft_consensus.cc:1226] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Rejecting Update request from peer d5b6839d9f5f4f6ab63bb9aabe75f4ba for earlier term 2. Current term is 3. Ops: []
I20240501 14:03:11.803439 18226 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.804256 18226 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.807292 17483 consensus_queue.cc:1046] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.807629 17483 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.807713 17483 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.807958 17483 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.808257 17483 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.813580 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.813819 15869 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.814339 15593 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.815414 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.815600 15866 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.815714 15866 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.816375 16130 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.819440 15803 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.820525 15866 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.822089 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.822393 16130 raft_consensus.cc:2379] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b in current term 3: Already voted for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in this term.
I20240501 14:03:11.822388 15866 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.823894 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.823920 18226 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.824113 15865 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.824267 15865 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.824877 18226 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.825451 18226 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.827525 15865 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.828548 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.828770 15999 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.828861 15999 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.829138 15999 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.829555 15999 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.830127 15866 raft_consensus.cc:1261] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.830763 17483 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.832298 15999 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.833005 15804 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.833351 17980 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.833498 17980 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.833794 17980 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.833925 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.833922 15865 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.834142 15736 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.835831 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.837848 15980 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:11.841050 15736 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.843303 15589 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.846271 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.846571 15736 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.849974 15736 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.850782 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.851011 15734 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.851159 15804 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.851729 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.851914 15736 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.854048 15734 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.854662 15736 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.854511 15591 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.854943 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.857363 15999 raft_consensus.cc:1261] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.857498 15999 pending_rounds.cc:76] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:11.857568 15999 pending_rounds.cc:97] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.857714 15999 raft_consensus.cc:2866] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.857767 15737 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.858309 15804 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.858530 15849 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:11.859550 17980 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.859721 17980 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.859807 18225 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.859985 18225 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.860087 15865 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.860018 17980 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.860188 15865 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.860369 18225 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.863020 18175 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.866396 15866 raft_consensus.cc:1261] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.866987 17483 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.873193 15865 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.873688 15867 raft_consensus.cc:1226] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Rejecting Update request from peer d5b6839d9f5f4f6ab63bb9aabe75f4ba for earlier term 3. Current term is 4. Ops: []
I20240501 14:03:11.874652 18175 consensus_queue.cc:1046] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.874959 18175 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.875051 18175 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.875391 18175 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.875783 18175 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.881420 15865 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.882169 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.884356 15591 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:11.885082 16130 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.884953 15591 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.885634 15803 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.886039 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.886276 15999 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.886859 18226 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.887027 15737 raft_consensus.cc:1261] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:03:11.887059 18226 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.887681 17483 consensus_queue.cc:1035] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.887516 18226 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.893793 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.894209 15865 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.894559 15865 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.895812 15999 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.899582 15865 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.900255 15865 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:11.900821 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.901068 15736 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.901813 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.902053 15999 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.902171 15999 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.902493 15999 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.903003 15999 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.903928 15736 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.904304 15804 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.905313 17980 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.905511 17980 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.905874 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.905871 17980 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:11.908321 15736 raft_consensus.cc:1261] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.908421 15736 pending_rounds.cc:76] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:11.908492 15736 pending_rounds.cc:97] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.908555 15736 raft_consensus.cc:2866] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.909132 18169 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.910517 15591 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.920318 15591 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:11.922423 17643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.923969 15999 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.931862 15865 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.932018 15865 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.934935 15716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.937806 15865 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.938346 15869 raft_consensus.cc:1226] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Rejecting Update request from peer d5b6839d9f5f4f6ab63bb9aabe75f4ba for earlier term 2. Current term is 3. Ops: []
I20240501 14:03:11.939260 17483 consensus_queue.cc:1046] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.939553 17483 raft_consensus.cc:3032] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.939623 17483 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.939865 17483 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.940191 17483 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.943379 15865 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.943719 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.943909 15737 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.944284 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.944545 16130 raft_consensus.cc:2396] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.946261 15737 raft_consensus.cc:2396] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:11.947566 15804 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.947875 18226 raft_consensus.cc:2726] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 14:03:11.948246 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.948724 15865 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.948930 15865 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.952095 15865 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.952936 15865 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.953450 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.953651 16130 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.953740 16130 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.954005 16130 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.954399 16130 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.955142 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.955394 15737 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.957194 16130 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.958052 15803 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:11.958420 18226 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.958561 18226 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.958688 15737 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.958848 18226 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.959498 18045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.963280 15999 raft_consensus.cc:1261] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:11.963393 15999 pending_rounds.cc:76] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:11.963475 15999 pending_rounds.cc:97] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:11.963539 15999 raft_consensus.cc:2866] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:11.964414 18175 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.964506 15591 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:11.966814 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.967020 15865 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.967147 15865 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.979902 15865 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.980566 15865 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:11.981261 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:11.981566 16130 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:11.981716 16130 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:11.981698 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.981894 15865 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.982023 15865 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.982213 16130 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.982663 16130 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.984817 15865 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.985726 16130 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:11.985926 15737 raft_consensus.cc:1261] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:11.986485 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.986663 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:11.986714 15734 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:11.986680 18169 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.986838 15737 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.992511 15734 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 4.
I20240501 14:03:11.992933 15804 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:11.993181 15737 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:11.993397 18226 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:11.993556 18226 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.993850 18226 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:11.994321 15865 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:11.994827 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:11.995054 15999 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:11.995158 15999 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:11.995421 15804 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:11.995431 15999 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.995801 15999 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.995859 17980 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:11.996014 17980 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:11.996250 17980 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:11.997756 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:11.997936 15865 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:11.998032 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:11.998289 17910 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.998591 15999 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:12.000347 15865 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.001346 17777 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.001358 15865 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.001902 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.001966 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.002118 15999 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:12.002300 15734 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:12.002233 15999 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.002451 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:12.002664 15869 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.002777 15869 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:12.002682 15999 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.003086 15999 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:12.005548 15734 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:12.005578 15869 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.006109 15999 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:12.006217 15804 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.006783 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.006862 18226 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:12.006892 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.006994 18226 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.006992 16130 raft_consensus.cc:3032] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:03:12.007074 15999 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:12.007109 16130 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.007314 18226 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.007427 16130 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.007745 16130 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:03:12.009884 15999 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 3.
I20240501 14:03:12.010537 16130 raft_consensus.cc:2396] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:12.011077 15869 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.013305 15804 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.013907 17980 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:03:12.013931 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:12.014070 17980 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.014179 15865 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.014328 15865 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.014353 17980 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.016650 15865 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.017612 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.017956 15999 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.018115 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.018314 16130 raft_consensus.cc:3032] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.018422 16130 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.018702 16130 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.019006 16130 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.021286 15999 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:12.021765 16130 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:12.022436 15803 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.022753 18226 raft_consensus.cc:2726] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:12.022926 15865 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.023176 15999 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:12.023378 15999 pending_rounds.cc:76] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:12.023972 15999 pending_rounds.cc:97] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:12.024061 15999 raft_consensus.cc:2866] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.024637 18169 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.033869 16108 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:12.034632 16130 raft_consensus.cc:1261] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:12.034916 16130 pending_rounds.cc:76] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 3
I20240501 14:03:12.035022 16130 pending_rounds.cc:97] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:12.035156 16130 raft_consensus.cc:2866] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.044585 15734 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:12.046070 17980 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.045168 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.046711 15737 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.046849 15737 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.046952 17483 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.054380 15737 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.060627 15591 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 3 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 4 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.063369 15999 raft_consensus.cc:1261] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:12.064147 18226 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.063460 15591 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.076149 15737 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.077035 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.077296 15999 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.077297 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.077353 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.077528 15865 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.077589 15737 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.077630 15865 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.077693 15737 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.077944 15865 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.078294 15865 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.076769 15591 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.078989 15591 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 2 to 3, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 3 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.080497 15999 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.080564 15737 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.081007 15672 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.081163 15865 raft_consensus.cc:2396] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.081364 18266 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.081486 15737 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.081544 18266 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.081792 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.081971 15869 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.082078 15869 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.081880 18266 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.082221 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.082440 15998 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.082324 15869 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.082664 15869 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.085134 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.085337 15734 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.085470 15734 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.085719 15869 raft_consensus.cc:2396] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.088510 15734 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.089789 15998 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.090426 15672 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:12.090857 17982 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.090981 17982 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.091209 17982 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.095146 15734 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.095624 16130 raft_consensus.cc:1261] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:12.095741 16130 pending_rounds.cc:76] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 3
I20240501 14:03:12.095816 16130 pending_rounds.cc:97] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:12.095893 16130 raft_consensus.cc:2866] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.096531 18226 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.095579 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.097488 16128 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.097465 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.098168 15737 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.098378 15737 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.100350 16128 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.100939 15734 raft_consensus.cc:1261] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:12.101112 15671 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.101544 18266 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.101529 18226 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.101706 18266 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.101918 15737 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.101980 18266 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.102260 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.102517 16000 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.102656 15737 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.106166 16000 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.108351 15589 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.109347 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.108897 15589 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.111028 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.111304 16000 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.109576 16128 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.115762 16128 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.116077 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.116273 15671 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.116446 15737 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.116595 17982 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.116664 15737 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.116761 17982 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.117077 17982 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.134487 16000 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.137064 15591 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.141865 15737 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.146340 15737 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.147168 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.147428 15865 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.147529 15865 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.147871 15865 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.148275 15865 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.151688 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.151933 16128 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.155344 16128 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.155814 15671 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.156142 18266 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.156303 18266 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.156569 18266 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.157078 15865 raft_consensus.cc:2396] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.157618 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.157840 15737 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.157981 15737 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.161222 15737 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.162283 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.162503 16128 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.179608 16128 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.180096 15671 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.180433 18273 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.180559 18273 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.180856 18273 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.183593 15734 raft_consensus.cc:1261] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.184284 18226 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.187810 16000 raft_consensus.cc:1261] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.188313 15999 raft_consensus.cc:1261] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:12.188843 18225 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.189023 18242 consensus_queue.cc:1035] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.197600 15737 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.197887 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.198273 15865 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.198385 15865 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.198701 15865 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.199015 15865 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.199335 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.199548 15737 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.199656 15737 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.200032 15591 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.201885 15865 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.202493 15737 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.203619 15737 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.203994 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.204303 16128 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.204931 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.205160 15865 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.205253 15865 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.205585 15865 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.205787 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.205981 15734 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.206127 15734 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.208336 16128 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.208710 15865 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.208724 15734 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.209478 15671 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.209754 17982 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.209885 17982 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.210171 17982 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.210708 15734 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.210750 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.210978 15999 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.211120 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.211294 15869 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.211383 15869 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.211601 15869 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.211869 15869 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.214243 15999 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.214591 15869 raft_consensus.cc:2396] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.214721 15672 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.214897 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.215067 15736 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.215188 15736 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.215415 18273 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.215553 18273 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.215797 18273 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.217281 15591 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.217943 15736 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.217945 15591 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.218570 15736 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.220014 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.220346 15999 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.222561 15865 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.225833 16130 raft_consensus.cc:1261] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:12.225947 16130 pending_rounds.cc:76] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 3
I20240501 14:03:12.226038 16130 pending_rounds.cc:97] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:12.226111 16130 raft_consensus.cc:2866] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.226972 18242 consensus_queue.cc:1035] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.231237 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.231487 15736 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.231586 15736 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.234398 15736 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.235534 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.235580 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.235769 16130 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.235898 15865 raft_consensus.cc:3032] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.236056 15865 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.238876 16130 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:12.244328 15999 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.246408 15865 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.252004 15865 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.252477 15589 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.253005 15589 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.257083 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.257334 16128 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.259886 16128 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.260190 15736 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.261571 15865 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:12.260407 15671 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:12.262071 15673 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.262208 17982 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.262367 18266 raft_consensus.cc:2726] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:03:12.262375 17982 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.262753 17982 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.265408 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.265579 15737 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.265677 15737 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.269980 15737 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.271786 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.272049 15737 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.272248 15591 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.272809 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.273329 16128 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.273483 16128 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.273689 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.273887 15737 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.273998 15737 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.274029 16128 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.274202 15999 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.274550 16128 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.278334 16128 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.278836 15671 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.279162 17982 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.279292 17982 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.279551 17982 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.286394 15589 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.294350 15737 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.295795 15999 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.297576 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.297856 16128 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.298863 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.299084 15999 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.300757 16128 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.301363 15671 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.301708 17982 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.301860 15999 raft_consensus.cc:2396] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.301862 17982 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.302129 17982 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.302317 15737 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.303866 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.304056 15736 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.304209 15736 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.307991 15736 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.309597 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.309842 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.312942 16000 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.313984 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.314225 15865 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.314330 15865 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.314427 15672 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.314766 18266 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.314651 15865 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.314884 18266 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.315054 15865 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.315129 18266 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.316282 15736 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.317703 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.317941 15737 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.318125 15737 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.318254 15865 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.321347 15737 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.322818 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.323026 15865 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.323150 15865 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.323479 15865 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.323761 15865 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.323531 15589 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.326870 15865 raft_consensus.cc:2396] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.329825 16128 raft_consensus.cc:1261] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.330596 18226 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.330735 15591 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.334805 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.335114 15999 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.337510 15737 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.338593 16000 raft_consensus.cc:1261] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.339493 18242 consensus_queue.cc:1035] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.340910 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.350849 15999 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.351327 15737 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.351567 15737 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.356209 15672 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:12.356555 15737 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.356738 17982 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.356899 17982 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.357331 17982 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.357491 15737 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.357968 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.358206 16128 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.359517 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.359733 15865 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.359812 15865 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.360119 15865 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.360407 15865 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.361617 16128 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.362159 15671 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.362509 17982 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:12.363224 15865 raft_consensus.cc:2396] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.363878 18273 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.364017 18273 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.364326 18273 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.364439 17982 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.365077 17982 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.366338 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" is_pre_election: true
I20240501 14:03:12.366595 16130 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.367063 15671 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.367419 18266 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 14:03:12.367538 18266 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:12.367636 18266 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.368798 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87" is_pre_election: true
I20240501 14:03:12.370687 18266 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.370651 15589 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.371992 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.372910 18266 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.373940 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.374145 16128 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.377259 16128 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.378137 15671 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:12.378196 15589 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.378630 18266 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.378775 18266 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.379040 18266 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.380039 15999 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.380877 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.381141 15737 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.381326 15737 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.384389 15737 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.385450 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.385697 15865 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.385993 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.388370 16128 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.389124 15865 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.390939 15673 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.391770 18266 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.391847 16128 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.392156 18266 raft_consensus.cc:686] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.392478 18266 consensus_queue.cc:237] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.392747 15589 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.393695 15737 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.395828 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.396039 15737 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.396178 15737 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.398526 15737 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.399114 15737 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.399590 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.399791 15865 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.399899 15865 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.400152 15865 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.400486 15865 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.401981 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.402253 16128 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.403138 15865 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.404072 15999 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.405375 16128 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.405647 18225 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.406577 15671 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.407029 18266 raft_consensus.cc:2726] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 14:03:12.405979 15737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.408915 15737 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.409098 15737 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.409926 15736 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.411123 18226 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.412230 15737 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.412364 16128 raft_consensus.cc:1261] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:03:12.412590 16128 pending_rounds.cc:76] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 3
I20240501 14:03:12.412706 16128 pending_rounds.cc:97] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:12.413161 15737 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.414440 15589 catalog_manager.cc:5484] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 2 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.415207 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.415423 15736 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.415541 15736 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.418709 15736 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.420722 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.421006 16000 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.433050 15998 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:03:12.434208 18226 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.435482 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.435739 15999 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.438721 15737 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:03:12.439677 18242 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.446960 16000 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.448379 15672 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.448815 18273 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.449018 18273 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.449318 18273 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.453318 15999 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.453687 15589 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.454586 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.454802 15865 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.454921 15865 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.455204 15865 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.455596 15865 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.458289 15736 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.412871 16128 raft_consensus.cc:2866] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.413461 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.459033 16130 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.459532 18242 consensus_queue.cc:1035] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.461869 16130 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.464442 15671 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:12.465171 18266 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.465363 18266 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.465662 18266 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.466068 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.466394 15736 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.466070 15999 raft_consensus.cc:1261] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:03:12.466643 15736 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.467080 15865 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 5.
I20240501 14:03:12.467089 18225 consensus_queue.cc:1035] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.453816 16000 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:03:12.469059 18225 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:03:12.474431 15737 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Refusing update from remote peer 42389da9ed2d4970ab94df10f7a2cb1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:03:12.481559 18225 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.484784 16000 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.485536 16130 raft_consensus.cc:1261] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.485751 17982 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.486879 17982 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.489818 15865 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 14:03:12.489943 15865 pending_rounds.cc:76] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.490037 15865 pending_rounds.cc:97] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 14:03:12.490180 15865 raft_consensus.cc:2866] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.490962 17982 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.527781 15736 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.545755 15865 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 14:03:12.545898 15865 pending_rounds.cc:76] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.545977 15865 pending_rounds.cc:97] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 14:03:12.546054 15865 raft_consensus.cc:2866] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.546514 15736 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.547255 18298 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:03:12.548538 15671 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.68:42895, user_credentials={real_user=slave}} blocked reactor thread for 49401.9us
I20240501 14:03:12.549391 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.549749 16128 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.549891 16128 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.550271 16128 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.550908 16128 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
W20240501 14:03:12.552395 15672 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.33.67:44773, user_credentials={real_user=slave}} blocked reactor thread for 53148us
I20240501 14:03:12.553406 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.553637 15999 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.553413 16000 raft_consensus.cc:1261] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.554121 15589 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 5 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.554694 16128 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.556674 15999 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.557150 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.557364 15736 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.557490 15736 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.557776 15671 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.558167 18294 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.558285 18294 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.558604 15999 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.558549 18294 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.560420 18273 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.560645 18273 consensus_queue.cc:1225] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer 09ee1930dbed4ac897d146bf885d0e87 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:12.562008 18273 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.563604 15736 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.563892 15999 pending_rounds.cc:76] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:12.564028 15999 pending_rounds.cc:97] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 14:03:12.564126 15999 raft_consensus.cc:2866] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.565075 15736 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.565217 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.565420 15865 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.565534 15865 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.565802 15865 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.566145 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.566524 15589 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.568346 15865 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.568773 15673 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.569317 18273 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.569485 18273 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.569734 18273 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.571040 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.571293 15736 raft_consensus.cc:465] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:03:12.572005 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.572233 15998 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.575258 15998 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.576303 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:12.576563 15736 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.576709 15736 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.577527 15589 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.584547 15736 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.585353 15736 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.585918 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.586149 15865 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.586313 15865 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.586613 15865 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.586997 15865 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.589139 15865 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.589567 15673 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.589872 18273 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.589978 18273 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.590248 18273 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.592990 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.593226 16000 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.593538 15589 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 4 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.598076 16000 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 4.
I20240501 14:03:12.612900 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.613205 15999 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.613361 15999 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.618649 16130 raft_consensus.cc:1261] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.620054 18297 consensus_queue.cc:1035] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.628873 15999 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.631918 15865 raft_consensus.cc:1261] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 14:03:12.632032 15865 pending_rounds.cc:76] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.632108 15865 pending_rounds.cc:97] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 14:03:12.632167 15865 raft_consensus.cc:2866] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.632786 18273 consensus_queue.cc:1035] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.633739 15999 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.633973 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.634227 16128 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.634233 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.634471 15736 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.634572 15736 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.634819 15736 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.635140 15736 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.636998 16128 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.637710 15934 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.638197 17846 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.638331 17846 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.638422 15736 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.638571 17846 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.644856 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.645398 15999 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.645741 15999 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.651057 15999 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.651840 15999 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.651683 15593 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.652238 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.652391 15865 raft_consensus.cc:1261] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.652448 15869 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.653462 18273 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.655196 15869 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.655642 15936 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.656009 17846 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.656296 17846 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.656632 17846 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.657377 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.657596 15736 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.657711 15736 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.657954 15736 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.658289 16128 raft_consensus.cc:1261] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.658341 15736 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.660071 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.660231 15999 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.660358 15999 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.664460 15999 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.664724 18273 consensus_queue.cc:1035] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240501 14:03:12.665426 16128 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.665653 15593 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.669521 15736 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.670991 18273 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240501 14:03:12.675540 15865 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 14:03:12.675668 15865 pending_rounds.cc:76] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.675740 15865 pending_rounds.cc:97] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 14:03:12.675827 15865 raft_consensus.cc:2866] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.676501 18297 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.682152 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.682510 15869 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.685525 15869 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.686167 15936 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.686561 17846 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.686703 17846 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.687062 17846 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.689149 15999 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.690270 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.690481 15736 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.690639 15736 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.690630 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.690934 15999 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.690918 15736 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.691089 15999 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.691344 15736 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.694229 15736 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.694315 15999 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.695190 15999 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.695747 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.695996 16130 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.696395 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.696570 16000 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.696692 16000 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.697196 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.697499 15869 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.697634 15869 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.698084 15869 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.698544 15869 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.698838 16130 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.699316 15934 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.699476 16000 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.699649 18155 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.699818 18155 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.700111 18155 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.700589 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.700788 15865 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.701462 15869 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.702656 15869 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 14:03:12.702827 15869 pending_rounds.cc:76] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.702550 15593 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.702975 15869 pending_rounds.cc:97] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 14:03:12.703105 15869 raft_consensus.cc:2866] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.703610 15865 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.703763 18297 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.704267 15936 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.704541 17846 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.704699 17846 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.704923 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.704941 17846 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.705152 16130 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.707895 16130 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.700116 16000 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.709327 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.709523 16000 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.709614 16000 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.721354 15999 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.721885 18273 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.724190 16000 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.725715 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.725908 16130 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.726441 15589 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 4 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.727157 15589 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 5, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.728461 16000 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.728533 16130 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.729216 15934 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.729842 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.730223 17846 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.730501 17846 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.730762 17846 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.730851 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.731101 15736 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.731228 15736 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.732869 15736 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.733465 15736 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.738940 16000 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.739140 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.740628 15736 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.743505 16000 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.743482 15589 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.744917 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.745206 15736 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.745105 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.745313 15736 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.745344 15869 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.745579 15736 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.746562 15736 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.748189 15869 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.748831 15936 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.749627 17846 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.749863 17846 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.750123 17846 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.751348 16000 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.753057 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.753248 16000 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.753342 16000 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.756114 16000 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.757164 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.757409 15869 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.758966 15865 raft_consensus.cc:1261] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 14:03:12.759063 15865 pending_rounds.cc:76] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.759130 15865 pending_rounds.cc:97] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 14:03:12.759191 15865 raft_consensus.cc:2866] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.760037 18273 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.760455 15869 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.760762 15869 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.761126 15869 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.761168 16130 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.762120 15736 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.763233 18273 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.764029 15869 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.764570 15936 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.764858 17846 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.764977 17846 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.765250 17846 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.767205 15593 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.768730 16000 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.771895 16128 raft_consensus.cc:1261] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.772073 15998 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.772182 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.772363 16129 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.772513 18297 consensus_queue.cc:1035] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.772604 18298 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.772796 18298 consensus_queue.cc:1225] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer 09ee1930dbed4ac897d146bf885d0e87 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:12.773747 15998 pending_rounds.cc:76] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:12.773845 15998 pending_rounds.cc:97] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 14:03:12.773923 15998 raft_consensus.cc:2866] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.775512 15999 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.776070 18298 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.775203 16129 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.778211 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.778489 16000 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.778687 16000 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.781581 16000 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.787863 16000 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.788544 15865 raft_consensus.cc:1261] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:12.790114 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.790117 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.790361 16130 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.790361 16000 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.790483 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.792250 18273 consensus_queue.cc:1035] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:12.792901 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.793509 16000 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.793880 15869 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.794741 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.796561 16130 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.799830 15869 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.797610 16129 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.800706 18298 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.803169 15936 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.803552 16000 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.803689 18155 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.803861 18155 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.804293 18155 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.806258 15589 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 4 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.806705 16000 raft_consensus.cc:1226] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 5. Current term is 6. Ops: []
I20240501 14:03:12.807281 18273 consensus_queue.cc:1046] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:12.807718 18273 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.807827 18273 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.808143 18273 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.808568 18273 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.809598 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.809814 16000 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.809919 16000 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.812872 16000 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.814199 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.814455 16129 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.816857 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.817215 15736 raft_consensus.cc:2396] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:12.817648 15935 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:12.817828 16129 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.818307 18155 raft_consensus.cc:2726] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:03:12.818552 15934 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.818773 16129 raft_consensus.cc:1226] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 5. Current term is 6. Ops: []
I20240501 14:03:12.818850 18155 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.819046 18155 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.819444 18155 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.820501 15998 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.821143 18273 consensus_queue.cc:1046] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.821628 18297 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.821717 18297 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.822021 18297 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.822304 18297 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.823900 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.825621 15736 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.825343 16000 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.824939 15593 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 5, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.827734 15998 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.828040 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.828238 16000 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.828384 16000 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.828529 18273 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.828820 18273 consensus_queue.cc:1225] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer 09ee1930dbed4ac897d146bf885d0e87 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:12.830164 15998 pending_rounds.cc:76] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:12.830289 15998 pending_rounds.cc:97] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 14:03:12.830456 15998 raft_consensus.cc:2866] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.831226 16000 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.832420 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.832715 15736 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.832880 15736 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.833369 15736 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.833905 15736 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.836143 16000 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.836926 15736 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.837597 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.837734 15935 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.837869 16130 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.838086 18155 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.838202 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.838295 18155 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.838413 16000 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.838536 16000 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.838608 18155 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.840759 16130 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.843123 16129 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 14:03:12.843581 16000 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.843737 18294 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.844286 16000 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.844058 15593 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.845108 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.845165 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.845343 15736 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.845461 15736 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.845520 16130 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.845780 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.845752 15736 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.845985 16000 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.846138 16000 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.846208 15736 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.849172 15736 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.849197 16130 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.849743 15935 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.850210 17846 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.850407 17846 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.850710 17846 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.854077 15593 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.856704 16000 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.857563 16000 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.858193 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.858440 15736 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.858551 15736 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.858772 16129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.858847 15736 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.859033 16129 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.859189 15736 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.862629 16129 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.863085 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.863380 16000 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.863468 15934 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.863618 16000 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.863794 18155 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.863938 18155 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.864212 18155 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.864362 15593 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.865307 16129 raft_consensus.cc:1261] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.866050 18294 consensus_queue.cc:1035] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.870174 15736 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.870915 15869 raft_consensus.cc:1261] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.871027 15998 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 14:03:12.871701 18273 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.872150 18273 consensus_queue.cc:1035] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.872782 16000 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.874011 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.874287 16130 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.879917 16130 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.880494 15934 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.880894 17846 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.882794 17846 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.883216 17846 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.884842 16000 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.885450 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.885723 15865 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.885824 15865 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.886134 15865 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.886521 15865 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.888605 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.889019 16000 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.889289 16000 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.889627 15865 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.892331 15869 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 14:03:12.892446 15869 pending_rounds.cc:76] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 4
I20240501 14:03:12.892509 15869 pending_rounds.cc:97] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 14:03:12.892590 15869 raft_consensus.cc:2866] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:12.892320 16000 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.893200 18273 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.893349 16000 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.894644 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.894845 15997 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.894968 15997 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.894927 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.896792 15997 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.897416 15997 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.897850 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.898066 16130 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.898311 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.898519 15737 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.898607 15737 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.898897 15737 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.899205 15737 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.900863 16130 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.901252 15737 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.901701 15935 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:12.902083 18155 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.902259 18155 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.902532 18155 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.903151 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.903354 15997 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.903455 15997 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.903277 15593 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.905818 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.906208 15736 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.906432 15736 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.907214 15998 raft_consensus.cc:1226] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 4. Current term is 5. Ops: []
I20240501 14:03:12.906831 15736 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
W20240501 14:03:12.908860 18273 consensus_queue.cc:1159] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "09ee1930dbed4ac897d146bf885d0e87" responder_term: 5 status { last_received { term: 4 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 14:03:12.909381 15736 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.909884 15593 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.910269 15593 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.911012 15997 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.911721 15997 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.913617 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.913813 15997 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.913944 15997 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.914194 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.914480 15866 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.916376 15997 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.917073 15997 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.918023 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.918035 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.918208 15734 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.918315 15734 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.918323 15865 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.918570 15734 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.918838 15734 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.921844 15734 raft_consensus.cc:2396] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:12.922005 15865 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 6.
I20240501 14:03:12.922166 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.922380 15997 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.922502 15997 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.922617 15936 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:12.923241 17846 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.923657 17846 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.924063 17846 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.924840 15997 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.925628 15997 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.926098 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.926282 16130 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.926908 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.927186 15737 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.927281 15737 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.927809 15736 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.928002 15593 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 6 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.928798 16130 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.929740 15866 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.929936 15934 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.930280 18155 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.930418 15935 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:12.930475 18155 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.930761 18333 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.930889 18333 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.930832 18155 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.931399 18333 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.932668 15936 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:12.933053 18333 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.933017 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.933212 18333 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.933223 15997 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.933400 15997 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.933274 15737 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.933473 18333 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.933571 15737 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.935120 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.935400 15736 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.935648 15736 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.935485 15593 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.936017 15736 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:12.936470 15737 raft_consensus.cc:2396] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 14:03:12.936142 15593 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:12.936647 15997 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.937248 15736 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.937770 15997 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.937891 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.938123 16130 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.938545 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.938733 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.938810 15865 raft_consensus.cc:3032] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:03:12.938927 16000 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.938932 15865 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:12.939020 16000 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.939436 15865 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.940021 15865 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:03:12.940889 16130 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.941593 16000 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.942217 16000 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:12.944176 15865 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 4.
I20240501 14:03:12.944895 15934 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.945246 17846 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:03:12.945441 17846 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.945767 17846 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.946161 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:12.946369 15998 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.946462 15998 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.948432 15998 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.949031 15998 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:12.949775 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:12.950024 16130 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.950284 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.950501 15737 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.950582 15737 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:12.950851 15737 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.951125 15737 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.952879 16130 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:12.953366 15736 raft_consensus.cc:2396] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.954806 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.954959 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.955060 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.955121 15866 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.955745 15737 raft_consensus.cc:2396] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 14:03:12.957609 15593 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 4, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 4 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.957996 15866 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.958197 15593 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:12.959311 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:12.959513 16130 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.958582 15865 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 5.
I20240501 14:03:12.959635 16130 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.961705 15936 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:12.962188 15936 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:12.962251 18155 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.962210 16130 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.962520 18155 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.962854 18333 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.962925 18155 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.963956 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.964241 15865 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.964442 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.964668 15997 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:12.964753 15997 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.965090 15997 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.965469 15997 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.968729 16130 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.969274 15865 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:12.970245 16066 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.970372 16129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:12.970572 16129 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.970666 16129 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:12.971634 15997 raft_consensus.cc:2396] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.973109 18333 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.973282 16129 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.973434 18333 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:12.974064 18169 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:12.974256 18169 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.974522 18169 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.975785 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:12.975997 15737 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:12.978013 15589 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.978286 15591 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:12.978673 15591 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 3 to 5, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 5 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.982452 16129 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:12.983150 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.983449 15997 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:12.983590 15997 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.984046 15997 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.984519 15997 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:12.984807 15737 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 7.
I20240501 14:03:12.985225 16129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:12.985352 16065 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:12.985450 16129 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.985563 16129 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.985675 17483 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:12.985811 17483 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.986094 17483 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:12.988700 16129 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.989429 15591 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 7 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:12.989825 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:12.989953 16129 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.990039 16000 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:12.990150 16000 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:12.990406 16000 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.990712 16000 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.990924 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:12.991108 16130 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.991209 16130 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:12.993778 16000 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:12.994415 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:12.994835 15865 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:12.997536 16130 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.997673 15865 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:12.998059 16066 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:12.998262 16130 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:12.998610 18169 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:12.998746 18169 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:12.999032 18169 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:12.999217 16129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:12.999387 16129 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.999482 16129 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.000360 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.000538 15866 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.002348 16129 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.002890 16129 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.003283 15866 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.003808 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.003998 16128 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.004101 16128 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.005347 16066 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.005676 17483 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.006340 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.006546 15737 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.006662 15737 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.006738 16128 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.006947 15737 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.007349 15737 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.007968 16128 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.008100 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.008312 15736 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.008884 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.009096 16128 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.009197 16128 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.009677 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.009868 15998 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.009953 15998 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.010218 15998 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.010519 15998 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.011086 15736 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.011688 16128 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.012688 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.012876 15736 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.012959 15736 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.013268 15736 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.013676 15736 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.013825 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.014030 15867 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.014909 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.015136 15866 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.015913 15591 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.016708 15736 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.016827 15867 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.017153 16065 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.017447 18169 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.017591 18169 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.017763 16128 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.017832 18169 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.019425 15998 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.019311 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.019598 16127 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.019766 16127 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.020957 15737 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.021241 17483 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.021489 17483 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.022578 16065 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:13.022811 17483 raft_consensus.cc:2726] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.023221 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.023389 16000 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.023459 16000 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.023519 16127 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.023641 16000 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.023872 16000 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.024796 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.025029 15867 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.025055 16127 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.025255 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.025411 15737 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.025491 15737 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.025794 15737 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.026197 15737 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.027571 15867 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.027962 16066 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.028333 17483 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.028455 17483 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.028784 15866 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.028692 17483 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.029189 16066 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.029543 18169 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.029656 18169 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.029872 18169 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.033419 16000 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.029162 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.033892 16130 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.034071 16130 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.034312 15593 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.034914 15593 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.035416 15593 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.035961 15593 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.037588 15737 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.038843 15997 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.052742 16130 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.053949 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.054183 15737 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.056942 15737 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 7.
I20240501 14:03:13.057416 16065 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.058658 18341 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.058903 18341 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.059355 18341 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.062091 16130 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.062225 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.062491 15997 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.062608 15997 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.062902 15997 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.063196 15997 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.062671 15592 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 7 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.065929 15997 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.068011 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.068251 16130 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.068387 16130 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.071456 16130 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.072571 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.072811 15865 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.073629 16130 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.075214 15865 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.075964 16066 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.075987 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.076215 15997 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.076305 15997 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.076295 18341 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.076467 18341 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.076640 15997 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.076731 18341 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.076931 15997 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.080302 15997 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.083861 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.084054 16130 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.084184 16130 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.087127 15592 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.087576 16130 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.089246 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.089283 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.089488 15997 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.089568 15997 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.089587 15737 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.089767 16130 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.090130 15997 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.090425 15997 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.091001 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.091269 16130 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.091420 16130 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.093225 15997 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.094272 16130 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.094516 15737 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.095069 16065 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:13.095073 16130 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.095384 18341 raft_consensus.cc:2726] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.095953 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.096233 15737 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.096236 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.096436 15865 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.099200 15737 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.099298 15865 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:13.099642 16065 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.099969 18341 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.100149 18341 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.100417 18341 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.101595 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.101830 16130 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.102066 16130 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.107067 16130 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.108243 16130 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.109072 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.109336 15997 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.109439 15997 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.109444 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.109652 16127 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.109759 16127 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.111258 15997 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.111572 15997 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.112797 16127 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.114758 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.115087 15736 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.117761 15997 raft_consensus.cc:2396] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.118809 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.119060 15736 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 7.
I20240501 14:03:13.120378 16065 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.120676 18169 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.120301 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.120826 18169 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.120947 15997 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.121043 15997 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.121146 18169 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.121310 15997 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.121568 15997 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.119514 16127 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.123210 15736 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.123469 16129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.123652 16129 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.123795 16129 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.123809 18155 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.126569 16129 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.128211 16129 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.128837 15865 raft_consensus.cc:1261] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.128911 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.129096 15734 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.130787 17846 consensus_queue.cc:1035] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.130869 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.131075 16000 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.131192 16000 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.131481 16000 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.131804 16000 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.131940 15734 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.134925 16128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.135214 16128 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.135378 16128 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.138253 16000 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.138185 16128 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.138906 16128 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.140641 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.140892 16126 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.141058 16126 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.141121 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.141330 15865 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.144171 15865 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.144191 16126 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.144680 16065 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:13.145012 17483 raft_consensus.cc:2726] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.145340 16126 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.146251 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.146590 16000 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.146760 16000 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.147140 16000 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.147522 16000 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.148461 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.149170 15734 raft_consensus.cc:3032] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.149616 15734 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.150231 15734 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.150403 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.150678 16126 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.150913 16126 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.151098 15734 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.154114 16126 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.154759 15736 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.155560 17846 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.156599 15734 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.119066 15737 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.158860 16065 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.159279 17483 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.159421 17483 raft_consensus.cc:686] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.159721 15865 raft_consensus.cc:1261] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.159730 17483 consensus_queue.cc:237] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.160280 18155 consensus_queue.cc:1035] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.161288 16000 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.163210 16126 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.164444 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.165194 16126 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.165688 16126 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.175330 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.175580 15734 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.175678 15734 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.177852 15997 raft_consensus.cc:2396] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.179066 15737 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.180509 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.180747 15865 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.182022 16066 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:13.182832 18169 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.183043 18169 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.183279 16126 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.183712 15865 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.183552 18169 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.184248 15980 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:03:13.184583 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.184787 15865 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.183743 16065 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:13.185864 18169 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.187222 15736 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.187503 15865 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.187984 18155 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.188225 15734 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.188464 18169 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.189126 15734 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.189103 16066 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.189438 17483 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.188787 18169 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.201417 16126 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.201517 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.201823 15866 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.202503 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.202736 16126 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.202852 16126 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.206163 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.206470 15737 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.206782 15866 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 14:03:13.208742 16126 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.207510 15734 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.210309 15865 raft_consensus.cc:1261] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.211298 18155 consensus_queue.cc:1035] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.212253 16126 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.213501 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.213721 15736 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.214432 15592 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 3 to 5, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.216580 15736 raft_consensus.cc:2396] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.218624 15737 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.219380 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.219692 15997 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.219817 15997 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.220253 15997 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.220785 16065 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.221966 18341 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.222175 18341 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.222620 18341 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.223692 15997 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.224749 15716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.229315 15997 raft_consensus.cc:2396] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.230383 16065 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:13.230887 18341 raft_consensus.cc:2726] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:03:13.231173 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.231336 16126 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.231524 16126 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.233783 16126 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.234755 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.234978 15865 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.236876 15865 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 5.
I20240501 14:03:13.237277 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.237383 16066 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.237674 18341 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:03:13.237819 18341 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.238076 18341 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.237512 15997 raft_consensus.cc:3032] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:03:13.238905 15997 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.239281 15997 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.239548 15997 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.242820 15849 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.242883 15997 raft_consensus.cc:2396] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.243131 16126 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 5 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.244359 16130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:13.244563 16130 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.244673 16130 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.247237 16130 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.247949 16130 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.248550 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.248816 15866 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.251565 15866 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 6.
I20240501 14:03:13.252601 17777 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.252867 16066 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.253278 18341 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.253449 18341 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.253731 18341 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.266491 18045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.268831 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.269374 15865 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.269627 15865 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.272645 15865 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.274011 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.274257 15737 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.275095 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.275337 15997 raft_consensus.cc:3032] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.275440 15997 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.276304 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.276487 16000 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.276580 16000 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.276837 16000 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.277292 16000 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.277439 15737 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.277906 15804 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.278263 18359 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.278411 18359 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.278694 18359 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.280499 16000 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.281810 15865 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.282068 15594 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 7 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.283095 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.283329 15866 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.283476 15866 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.285890 15866 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.288462 15866 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.288856 15997 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.289237 15997 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.289490 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.289688 15866 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.289796 15866 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.291301 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.291504 16000 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.294610 16000 raft_consensus.cc:2396] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.294658 15997 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.296898 15866 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.298121 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.298345 15737 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.301147 15737 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.301640 15804 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.301956 18225 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.302104 18225 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.302465 18225 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.302901 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.303216 15866 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.305028 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.305240 15865 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.305343 15865 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.308914 15865 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.310086 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.310218 15865 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.310340 15997 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.311038 17910 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.311300 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.311519 15865 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.311638 15865 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.313419 15997 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.314793 15865 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.315361 15865 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.315579 16000 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.315685 16000 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.316057 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.316269 15737 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.316571 15866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.316740 15866 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.317165 16000 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.317569 16000 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.318701 15594 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 7 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.319536 15737 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.319916 15804 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.316851 15866 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.320197 18359 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.320358 18359 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.320621 18359 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.322681 15866 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.323927 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.324087 16000 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.324255 15997 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.324692 15866 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.325701 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.325924 15865 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.326045 15865 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.327625 15997 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.328836 15865 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.329586 15865 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.330225 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.330487 15737 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.331887 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.332124 16127 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.332226 16127 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.332499 16127 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.332847 16127 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.333151 17483 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.333515 15737 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.333436 17483 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.333943 15804 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.334256 18359 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.334393 18359 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.334656 18359 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.335914 16127 raft_consensus.cc:2396] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.337699 15592 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.338081 15592 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.338466 15592 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 7 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.338743 15592 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.338995 15592 catalog_manager.cc:5484] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.340443 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.340653 16130 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.340754 16130 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.341020 16130 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.341357 16130 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.344542 16108 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:03:13.345005 17643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.345264 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.345499 16128 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.345603 16128 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.345836 16128 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.346154 16128 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.349932 16130 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.353072 15803 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.353540 18225 raft_consensus.cc:2726] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.354205 16128 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.354626 15803 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.355005 18225 raft_consensus.cc:2726] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:03:13.356678 15592 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 6, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.357388 15592 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 6 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.357946 15592 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 4 to 5, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 5 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.365409 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.365701 16126 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.365873 16126 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.366257 16126 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.366678 16126 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.367208 15997 raft_consensus.cc:1261] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20240501 14:03:13.367350 15997 pending_rounds.cc:76] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 5
I20240501 14:03:13.367408 15997 pending_rounds.cc:97] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 14:03:13.367493 15997 raft_consensus.cc:2866] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.368211 17483 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.384577 15594 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.388810 16126 raft_consensus.cc:2396] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.389497 15803 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.389462 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.389771 16000 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.389875 16000 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.390177 18225 raft_consensus.cc:2726] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:03:13.391444 15737 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.397545 15737 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.398701 18341 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.392127 18155 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240501 14:03:13.407061 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.407353 15865 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.390162 16000 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.407855 15865 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.408318 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.410899 15736 raft_consensus.cc:1261] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:13.411507 17483 consensus_queue.cc:1035] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.414580 15866 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.415736 17483 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.415956 15866 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:03:13.420636 15866 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.421314 18337 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.437438 15734 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:13.438586 18361 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.440706 16000 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.441203 15594 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.448590 16000 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20240501 14:03:13.448751 16000 pending_rounds.cc:76] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 5
I20240501 14:03:13.448825 16000 pending_rounds.cc:97] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 14:03:13.448899 16000 raft_consensus.cc:2866] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.449561 18169 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.451376 15865 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.452220 15865 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.452761 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.452960 16000 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.453197 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.453398 15865 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.453514 15865 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.453711 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.454066 16126 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.454195 16126 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.454540 16126 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.454878 16126 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.455999 16000 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.458083 15865 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.458771 15865 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.459475 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.459708 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.459900 15865 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.459929 16126 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.460040 15865 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.460786 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.461076 16000 raft_consensus.cc:2379] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b in current term 6: Already voted for candidate 09ee1930dbed4ac897d146bf885d0e87 in this term.
I20240501 14:03:13.461831 15803 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.462288 18225 raft_consensus.cc:2726] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.462368 15734 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.462772 15804 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:13.463177 18225 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.463357 18225 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.463593 18225 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.465756 15865 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.466698 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.466902 15734 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.466985 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.467213 16126 raft_consensus.cc:3032] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.467298 16126 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.467012 15594 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.467589 16126 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.467911 16126 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.470217 15734 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.470625 15804 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.470960 18225 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.471113 18225 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.471386 18225 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.472677 15866 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.473457 18337 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.473763 15865 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.473974 16126 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.474939 15594 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.475826 15867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.476011 15867 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.476140 15867 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.479112 15865 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.479405 15867 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.480273 18337 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.481081 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.481315 16126 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.481410 16126 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.481556 15867 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.481689 16126 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.482056 16126 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.482699 15867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.482939 15867 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.483136 15867 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.484964 16126 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.485358 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.485572 15736 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.485808 15803 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.488219 15736 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.488549 18225 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.488714 18225 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.488942 18225 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.489485 15734 raft_consensus.cc:1261] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240501 14:03:13.489593 15734 pending_rounds.cc:76] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 5
I20240501 14:03:13.489655 15734 pending_rounds.cc:97] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240501 14:03:13.489743 15734 raft_consensus.cc:2866] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.490969 17846 consensus_queue.cc:1035] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.491245 15867 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.492170 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.492383 16126 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.492462 16126 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.492820 15734 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240501 14:03:13.492736 16126 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.493065 15734 pending_rounds.cc:76] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 5
I20240501 14:03:13.493256 15737 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240501 14:03:13.493420 15737 pending_rounds.cc:76] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 5
I20240501 14:03:13.493479 15737 pending_rounds.cc:97] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240501 14:03:13.493556 15737 raft_consensus.cc:2866] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.493863 16126 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.493252 15734 pending_rounds.cc:97] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240501 14:03:13.494274 15734 raft_consensus.cc:2866] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.494261 18337 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.496323 17846 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.496608 16126 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.497885 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.498137 15737 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.501283 15737 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.502461 15804 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.502777 18359 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.502964 18359 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.503242 18359 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.505666 15866 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 14:03:13.506212 18333 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.506532 15867 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.508463 15869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.510665 15869 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.510794 15869 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.513690 15869 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.514421 15869 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.514865 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.515095 16000 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.515203 16000 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.515462 16000 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.515648 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.515818 16000 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.515844 15734 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.516542 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.516763 15865 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.516927 15865 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.518630 16000 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.518878 15734 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.519795 15865 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.520001 15594 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.520493 15865 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.520864 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.521190 16126 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.521200 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.521077 15804 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.521395 15997 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.521420 16126 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.521927 16126 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.522006 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.522277 15865 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.522328 16126 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.522413 15865 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.524173 15997 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.524592 18225 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.524778 18225 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.525199 18225 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.525494 16126 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.526118 15865 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.526448 15594 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.526942 15865 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.527949 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.528168 15734 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.528263 15804 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.529048 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.529218 15865 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.529172 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.529333 15865 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.529552 15997 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.529676 15997 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.529976 15997 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.530355 15997 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.530972 15734 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.532524 15997 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.532819 18225 raft_consensus.cc:2726] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.532920 15865 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.533177 15804 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.534224 15865 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.534368 18225 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.534597 18225 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.534940 18225 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.535025 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.535434 15734 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.536592 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.536425 15592 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 6 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 7 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.536904 16126 raft_consensus.cc:3032] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.537072 16126 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.537395 16126 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.537729 16126 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.538893 15734 raft_consensus.cc:2454] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.539242 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.539464 15865 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.539600 15865 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.540472 16126 raft_consensus.cc:2396] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.541476 15804 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.542049 18225 raft_consensus.cc:2781] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.542212 18225 raft_consensus.cc:686] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.542470 18225 consensus_queue.cc:237] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.542757 15865 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.542779 15592 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.543319 15865 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.543962 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.544165 16126 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.544246 16126 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.544517 16126 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.544826 16126 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.544754 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.544946 15734 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.545158 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.545296 15865 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.545378 15865 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.547181 15865 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.547776 16126 raft_consensus.cc:2396] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.547807 15734 raft_consensus.cc:2454] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.547827 15865 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.548218 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.548403 15804 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.548410 15734 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.548581 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.548683 18359 raft_consensus.cc:2781] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.548784 16127 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.548825 18359 raft_consensus.cc:686] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.548880 16127 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.549156 18359 consensus_queue.cc:237] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.549378 16127 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.549666 16127 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.549455 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.549933 15865 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.550048 15865 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.552405 16127 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.553196 15734 raft_consensus.cc:2454] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 7.
I20240501 14:03:13.553619 15804 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.554015 18225 raft_consensus.cc:2781] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.554149 18225 raft_consensus.cc:686] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.554286 15865 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.554402 18225 consensus_queue.cc:237] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.555101 15865 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.555923 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.556108 15997 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.556193 15997 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.556337 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.556452 15997 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.556546 15865 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.556681 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.556805 15997 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.559303 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.559500 15737 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.559639 15997 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.560592 15804 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.560547 15865 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.560878 18359 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.561383 15865 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.561857 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.562047 15736 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.562311 15737 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.564752 15736 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.564880 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.565124 15997 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.565233 15997 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.565254 15804 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.565527 15997 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.565896 15997 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.567749 18225 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:03:13.568001 18225 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.568792 18225 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.569137 18359 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Leader. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.569427 18359 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.570266 15592 catalog_manager.cc:5484] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 6 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 7 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.570827 15592 catalog_manager.cc:5484] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 6 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 7 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.571355 15592 catalog_manager.cc:5484] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.573277 15997 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42389da9ed2d4970ab94df10f7a2cb1b in term 6.
I20240501 14:03:13.573941 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.574155 15865 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.574265 15865 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.576470 15865 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.577461 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.577654 15997 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.577687 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.577843 16127 raft_consensus.cc:3032] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:03:13.577908 16127 raft_consensus.cc:729] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.578122 16127 consensus_queue.cc:260] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.578334 16127 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:03:13.581005 15997 raft_consensus.cc:2396] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.581214 16127 raft_consensus.cc:2396] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:03:13.580925 15592 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.581529 15592 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b reported cstate change: term changed from 5 to 6, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66). New cstate: current_term: 6 leader_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.581681 15803 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.582073 15865 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 6 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.582170 18385 raft_consensus.cc:2726] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:03:13.584008 15865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
from {username='slave'} at 127.0.0.1:45692
I20240501 14:03:13.584218 15865 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.584350 15865 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.586565 15865 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.587543 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.587553 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.587752 16127 raft_consensus.cc:3032] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.587879 15997 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.587886 16127 raft_consensus.cc:729] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.588335 16127 consensus_queue.cc:260] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.588713 16127 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.590977 15997 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.591670 16127 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42389da9ed2d4970ab94df10f7a2cb1b for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.592041 15803 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b; no voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.592305 15865 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.592342 18385 raft_consensus.cc:2726] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.602443 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.602695 15734 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.602835 15734 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.605679 15734 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.606392 15734 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.606850 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.607075 15865 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.607158 15865 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.607527 15865 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.607828 15865 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.610889 15865 raft_consensus.cc:2396] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.611415 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.611624 15734 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.611732 15734 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.614112 15734 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.615288 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.615305 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.615505 15865 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.615512 16000 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.615579 15865 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.615860 15865 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.616127 15865 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.617148 15734 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.617789 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.617981 15997 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.618358 16000 raft_consensus.cc:2454] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.618788 15865 raft_consensus.cc:2396] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.619005 15672 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.619405 18370 raft_consensus.cc:2781] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.619560 18370 raft_consensus.cc:686] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.619866 18370 consensus_queue.cc:237] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.620739 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.621367 15734 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.621491 15734 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.623020 15592 catalog_manager.cc:5484] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.624531 16000 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240501 14:03:13.624648 16000 pending_rounds.cc:76] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 5
I20240501 14:03:13.624701 16000 pending_rounds.cc:97] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240501 14:03:13.624774 16000 raft_consensus.cc:2866] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.625368 18169 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.626698 15734 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.627880 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.628082 16127 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.628173 16127 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.628428 16127 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.628788 16127 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.630308 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.630486 15998 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.631601 15997 raft_consensus.cc:2454] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.631726 16127 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.632220 15671 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.632539 18273 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.632673 18273 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.632907 18273 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.635761 15734 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.637118 15734 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 14:03:13.638069 18361 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.639397 15672 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.639791 18370 raft_consensus.cc:2781] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.639951 18370 raft_consensus.cc:686] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.640198 18370 consensus_queue.cc:237] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.642962 15998 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.643143 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.643389 15736 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.643492 15736 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.646371 15736 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.647403 15592 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.648020 15592 catalog_manager.cc:5484] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.652493 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.652720 16127 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.655602 16127 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.656225 15671 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.656915 18273 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.657080 18273 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.657399 18273 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.659664 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.659906 16000 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.662892 16000 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.665073 15736 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.666816 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.667451 15736 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.667604 15736 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.667582 15592 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 8, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.672741 15736 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.674091 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.674351 15865 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.674454 15865 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.674427 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.674736 16127 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.674798 15865 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.675101 15865 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.677822 16127 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.677989 15865 raft_consensus.cc:2396] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.678349 15671 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.678654 18273 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.678823 18273 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.679066 18273 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.681079 15736 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.682228 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.682098 15592 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.682435 15736 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.682540 15736 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.685266 15736 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.686465 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.686708 16127 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.688122 15736 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.688215 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.688570 15865 raft_consensus.cc:3032] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.688717 15865 raft_consensus.cc:729] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.689096 15865 consensus_queue.cc:260] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.689321 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.689491 15865 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.689515 15736 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.689627 15736 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.692386 15865 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.692413 16127 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.692909 15673 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.693295 18370 raft_consensus.cc:2726] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.693675 15736 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.694717 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.694934 16127 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.697604 16127 raft_consensus.cc:2396] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.698516 15736 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.698781 15734 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:13.698952 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.699137 15865 raft_consensus.cc:3032] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.699431 15865 raft_consensus.cc:729] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.699455 17483 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.699728 15865 consensus_queue.cc:260] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.699993 15865 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.699940 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.700146 15734 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.700263 15734 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.702663 15734 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.704195 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.704388 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.704512 15998 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer d5b6839d9f5f4f6ab63bb9aabe75f4ba: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20240501 14:03:13.704648 15998 pending_rounds.cc:76] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 5
I20240501 14:03:13.704708 15998 pending_rounds.cc:97] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 14:03:13.704789 15998 raft_consensus.cc:2866] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:13.705497 15865 raft_consensus.cc:2396] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.705590 17483 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.706284 15673 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.706560 18370 raft_consensus.cc:2726] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.707098 16000 raft_consensus.cc:2454] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.707496 15672 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.707791 18370 raft_consensus.cc:2781] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.707938 18370 raft_consensus.cc:686] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.707999 15734 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.708192 18370 consensus_queue.cc:237] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.709993 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.710400 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.710561 15734 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.710682 15734 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.711194 15592 catalog_manager.cc:5484] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.712255 15865 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.712360 15865 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.712607 15865 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.712857 15865 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.713186 15734 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.713973 15734 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.714637 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.714872 15998 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.714993 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.715333 15734 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.715435 15734 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.717038 15865 raft_consensus.cc:2396] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.717227 15734 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.717578 15998 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.717835 15734 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.718334 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.718405 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.718537 16126 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.718612 15865 raft_consensus.cc:3032] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.718693 15865 raft_consensus.cc:729] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.718936 15865 consensus_queue.cc:260] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.719262 15865 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.720803 16126 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.721660 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.721823 15734 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.721940 15865 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:13.721944 15734 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.722402 15673 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.722910 18370 raft_consensus.cc:2726] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.724066 15734 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.724668 15734 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.725180 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.725508 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.725585 15998 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.725692 16126 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.725781 16126 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.726536 16126 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.726861 16126 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.728614 16126 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.728958 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.729101 15671 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.729209 15734 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.729224 15998 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.729310 15734 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.729465 18370 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.729619 18370 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.729684 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.729873 16127 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.729943 18370 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.731750 15734 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.732378 15734 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.732429 16127 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.732786 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.733024 16126 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.733017 15671 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:13.733125 16126 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.733361 16126 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.733240 15592 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.733738 16126 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.734301 18273 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.734447 18273 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.734730 18273 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.736166 16126 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.736160 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.736414 15998 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.736868 15671 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.737135 18370 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.737555 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.737717 15734 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.737833 15734 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.739728 15734 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.739893 15998 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.740785 15734 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.741042 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.741269 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.741917 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.742121 15734 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.742233 15734 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.744117 16000 raft_consensus.cc:2454] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.744606 15672 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.745126 15734 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.745826 18370 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.746083 18400 raft_consensus.cc:2781] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.746218 18400 raft_consensus.cc:686] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.746176 18370 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.746763 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.747020 15869 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.747148 15869 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.747488 15869 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.747870 15869 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.751833 15869 raft_consensus.cc:2396] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.752863 15734 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.754243 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.754498 15734 raft_consensus.cc:482] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.754655 15734 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.756201 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.756419 15865 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.756495 15865 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.756789 15865 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.757078 15865 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.758316 15734 raft_consensus.cc:504] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.759740 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.760015 16126 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.746484 18400 consensus_queue.cc:237] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.763849 16126 raft_consensus.cc:2396] T d773c0f7122f47fbbb79970f00ae137f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.765511 15734 leader_election.cc:288] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.766744 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.767200 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d773c0f7122f47fbbb79970f00ae137f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.767421 15869 raft_consensus.cc:3032] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.767530 15869 raft_consensus.cc:729] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.767776 15869 consensus_queue.cc:260] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.768069 15869 raft_consensus.cc:3037] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.768813 15592 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 8, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.767011 15734 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.770341 15734 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.770844 15869 raft_consensus.cc:2396] T d773c0f7122f47fbbb79970f00ae137f P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.771453 15673 leader_election.cc:302] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.771996 18273 raft_consensus.cc:2726] T d773c0f7122f47fbbb79970f00ae137f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 14:03:13.772866 15865 raft_consensus.cc:2396] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.772984 15734 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.774492 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.774714 16126 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.775420 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.775620 15998 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.777971 16126 raft_consensus.cc:2396] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.778350 15998 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.779068 15672 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.779605 18401 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.779471 15592 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.780007 18401 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.779989 15592 catalog_manager.cc:5484] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.780406 18401 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.781524 15734 leader_election.cc:288] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.781615 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.781821 15865 raft_consensus.cc:3032] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.781888 15865 raft_consensus.cc:729] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.782158 15865 consensus_queue.cc:260] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.782475 15865 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.782541 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.782756 15734 raft_consensus.cc:482] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.782874 15734 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.785893 15734 raft_consensus.cc:504] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.786844 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.786864 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a2f1fb5d64858a323a0c2dd9eab5c" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.787019 16126 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.787034 15869 raft_consensus.cc:3032] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.787115 15869 raft_consensus.cc:729] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.787344 15869 consensus_queue.cc:260] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.787593 15869 raft_consensus.cc:3037] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.788940 16126 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:13.789657 15734 leader_election.cc:288] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.790349 15869 raft_consensus.cc:2396] T b99a2f1fb5d64858a323a0c2dd9eab5c P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:03:13.790149 15594 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.790663 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.790848 15734 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.790948 15734 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.791246 15673 leader_election.cc:302] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.791393 15865 raft_consensus.cc:2396] T 2b3735e79e334359b05c013508d99f3c P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.791795 15673 leader_election.cc:302] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.792188 18273 raft_consensus.cc:2726] T b99a2f1fb5d64858a323a0c2dd9eab5c P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 14:03:13.792317 18273 raft_consensus.cc:2726] T 2b3735e79e334359b05c013508d99f3c P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 14:03:13.793709 15734 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.794821 15734 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.794761 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.794972 16126 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.795802 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.795966 15734 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.796082 15734 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.798058 16126 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.798467 15671 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.798808 18401 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.798867 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.799108 15998 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.799425 18401 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.799733 18401 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.802233 15998 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.803902 15734 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.805356 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.805596 15998 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.810191 15998 raft_consensus.cc:2454] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.810856 15672 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.811254 18273 raft_consensus.cc:2781] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.811435 18273 raft_consensus.cc:686] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.811556 15734 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.811772 18273 consensus_queue.cc:237] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.811854 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.812079 15865 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.812147 15865 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.812400 15865 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.812651 15865 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.813748 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.814008 15734 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.814147 15734 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.816232 15865 raft_consensus.cc:2396] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.818130 15734 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.819499 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.819729 16126 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.819967 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.820191 15998 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.823097 15998 raft_consensus.cc:2396] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.823616 16126 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.823678 15734 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.824311 15671 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:13.824594 18273 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.824754 18273 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.825073 18273 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.825225 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.825394 15734 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.825532 15734 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.829207 15734 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.830132 15734 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.830514 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.830725 15865 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.830796 15865 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.831051 15865 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.831326 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.831997 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.832211 15998 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.832252 15736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.832464 15736 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.832597 15736 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.835561 15998 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.835783 15594 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 8, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.836057 15672 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.836061 15736 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.836429 18273 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.836561 18273 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.836769 15736 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:13.836830 18273 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.838224 15865 raft_consensus.cc:2396] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.839010 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.839305 16000 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.842483 16000 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.842892 15672 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:13.843202 18401 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.843408 18401 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.843726 18401 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.845970 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.846252 16126 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.846475 16126 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:13.846899 16126 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.847342 16126 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.847129 15594 catalog_manager.cc:5484] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.847910 15594 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.848497 15594 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 5 to 7, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.849164 15734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
from {username='slave'} at 127.0.0.1:49634
I20240501 14:03:13.849385 15734 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.849499 15734 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.849143 15594 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 7 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 8 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:13.850728 16126 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 8.
I20240501 14:03:13.855389 15734 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.856902 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.857149 15865 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:03:13.857223 15865 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42389da9ed2d4970ab94df10f7a2cb1b, State: Running, Role: LEADER
I20240501 14:03:13.857527 15865 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.857813 15865 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.860661 15865 raft_consensus.cc:2396] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 28461d3f709145e29fc3d3d1c7cce205 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.861557 15734 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.862080 16000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "28461d3f709145e29fc3d3d1c7cce205" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:13.862366 16000 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.866119 16000 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28461d3f709145e29fc3d3d1c7cce205 in term 7.
I20240501 14:03:13.867659 15672 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.868119 18401 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.868335 18401 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Leader. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.868691 18401 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.871990 15594 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 reported cstate change: term changed from 6 to 7, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65). New cstate: current_term: 7 leader_uuid: "28461d3f709145e29fc3d3d1c7cce205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:13.892505 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.892745 16000 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.892871 16000 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.895649 16000 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.896803 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.897047 16000 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.897109 16126 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.897616 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.897804 15734 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:13.897891 15734 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.897962 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.898185 15998 raft_consensus.cc:482] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.898159 15734 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.898281 15998 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.898481 15734 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.899834 16126 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:13.900394 15998 raft_consensus.cc:504] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.900857 15734 raft_consensus.cc:2396] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 14:03:13.900873 15998 leader_election.cc:288] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.901335 15934 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:13.901618 17846 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:13.901577 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.901782 17846 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.901819 15865 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.902089 17846 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:13.902385 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20717d395e98428594d60999df42298f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.902652 16000 raft_consensus.cc:482] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.902779 16000 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.906853 16000 raft_consensus.cc:504] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.908282 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.908546 15869 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.911971 15869 raft_consensus.cc:2396] T 20717d395e98428594d60999df42298f P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.914767 16000 leader_election.cc:288] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:13.916095 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.916383 15998 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.916541 15998 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.901369 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c03daaf6d13485c96f88449e2b18fce" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.915095 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20717d395e98428594d60999df42298f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.920468 15737 raft_consensus.cc:3032] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:13.920603 15737 raft_consensus.cc:729] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.921010 15865 raft_consensus.cc:2396] T 6c03daaf6d13485c96f88449e2b18fce P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.921522 15737 consensus_queue.cc:260] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.922005 15737 raft_consensus.cc:3037] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.925027 15736 raft_consensus.cc:3032] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:13.925199 15736 raft_consensus.cc:729] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.925582 15736 consensus_queue.cc:260] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:13.926012 15736 raft_consensus.cc:3037] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.930677 15737 raft_consensus.cc:2396] T 20717d395e98428594d60999df42298f P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.931376 15935 leader_election.cc:302] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.931993 18333 raft_consensus.cc:2726] T 20717d395e98428594d60999df42298f P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 14:03:13.933482 15736 raft_consensus.cc:2396] T 6c03daaf6d13485c96f88449e2b18fce P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:13.934116 15935 leader_election.cc:302] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:13.934672 18333 raft_consensus.cc:2726] T 6c03daaf6d13485c96f88449e2b18fce P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 14:03:13.935863 15998 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.937057 15592 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 9 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:13.938211 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.938443 15865 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.939313 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.939585 16126 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:13.941177 15865 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 7.
I20240501 14:03:13.942385 15936 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:13.943604 16126 raft_consensus.cc:2396] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:13.943646 17846 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:13.944325 17846 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.944514 15998 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.944612 17846 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.946238 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.946748 15998 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.947577 15998 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.949980 15592 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 7 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:13.956422 15998 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.957693 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.957903 16126 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.960636 16126 raft_consensus.cc:2396] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:13.961313 15998 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:13.961920 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.962179 15865 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.962517 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.962724 15998 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.962852 15998 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.965035 15865 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:13.965433 15936 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:13.965703 17846 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:13.965894 17846 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.966018 15998 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.966207 17846 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.967185 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.967459 15736 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:13.967571 15736 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.967887 15736 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.968284 15736 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.969846 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:13.970118 16126 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:13.980154 15736 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 14:03:13.981155 16126 raft_consensus.cc:2454] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:13.981832 15934 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:13.982384 18333 raft_consensus.cc:2781] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:13.982614 18333 raft_consensus.cc:686] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:13.983053 18333 consensus_queue.cc:237] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:13.985121 15998 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:13.987246 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:13.987541 16000 raft_consensus.cc:482] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:13.987740 16000 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.991065 16000 raft_consensus.cc:504] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.994179 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:13.994545 15736 raft_consensus.cc:3032] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:13.994724 15736 raft_consensus.cc:729] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:13.995136 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434887fc6944dd587f54b28fbe73447" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:13.995167 15736 consensus_queue.cc:260] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:13.995435 15865 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.995734 15736 raft_consensus.cc:3037] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:13.999336 15865 raft_consensus.cc:2396] T 9434887fc6944dd587f54b28fbe73447 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:13.999825 15736 raft_consensus.cc:2396] T 9434887fc6944dd587f54b28fbe73447 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:14.000401 15935 leader_election.cc:302] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:14.000814 17846 raft_consensus.cc:2726] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 14:03:14.001365 16000 leader_election.cc:288] T 9434887fc6944dd587f54b28fbe73447 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.017982 18045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.025914 15593 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 6 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.031834 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.032058 16000 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.032250 16000 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.035349 16000 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.036043 15593 catalog_manager.cc:5484] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 9 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:14.036721 16000 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:14.057242 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.057576 16126 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.062216 16126 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:14.064245 15980 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:14.106400 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.106774 16000 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.106949 16000 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.115793 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.116137 15865 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.121013 15865 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 7.
I20240501 14:03:14.121660 15936 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.122040 17846 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:14.122247 17846 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.122583 17846 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.124274 16000 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.133103 16000 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.127450 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.133335 16126 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.134858 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.135380 15865 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.135838 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.136128 16000 raft_consensus.cc:482] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.136305 16000 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.144531 15716 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:03:14.145262 17777 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.146962 15998 raft_consensus.cc:1261] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 14:03:14.155117 15593 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 7 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.159332 18401 consensus_queue.cc:1035] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.159575 18401 consensus_queue.cc:1225] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer 09ee1930dbed4ac897d146bf885d0e87 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:14.161041 15997 pending_rounds.cc:76] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 3
I20240501 14:03:14.161160 15997 pending_rounds.cc:97] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 14:03:14.161231 15997 raft_consensus.cc:2866] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.161329 15997 pending_rounds.cc:97] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240501 14:03:14.161386 15997 raft_consensus.cc:2866] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.152438 16000 raft_consensus.cc:504] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.166074 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.166282 16000 leader_election.cc:288] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.166350 15869 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.169811 16000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.169878 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8331fc2f5f48278928267c260356c8" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.170117 15736 raft_consensus.cc:3032] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.170239 15736 raft_consensus.cc:729] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.170564 15736 consensus_queue.cc:260] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.170888 15736 raft_consensus.cc:3037] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.174345 15736 raft_consensus.cc:2396] T 4a8331fc2f5f48278928267c260356c8 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:14.181708 15869 raft_consensus.cc:2396] T 4a8331fc2f5f48278928267c260356c8 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:14.182790 15936 leader_election.cc:302] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:14.183075 16127 raft_consensus.cc:1261] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 14:03:14.183496 17846 raft_consensus.cc:2726] T 4a8331fc2f5f48278928267c260356c8 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 14:03:14.184013 18401 consensus_queue.cc:1035] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.187848 16126 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:14.187850 16000 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.188009 16000 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.190884 16000 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.191459 15865 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.193814 16128 raft_consensus.cc:1261] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 14:03:14.194835 16127 raft_consensus.cc:1261] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 14:03:14.195039 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.197592 15936 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.198068 17846 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.198140 15999 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 14:03:14.198235 17846 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.198493 17846 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.199605 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.199831 16128 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.202843 16128 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:14.203526 18273 consensus_queue.cc:1035] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.204028 18273 consensus_queue.cc:1035] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.205237 18400 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.205377 16000 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.206079 15934 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.207038 18337 raft_consensus.cc:2781] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.207238 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.207237 18337 raft_consensus.cc:686] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.207480 15999 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.207588 15999 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.207684 18337 consensus_queue.cc:237] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.209992 16127 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 14:03:14.210403 15999 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.210896 15865 raft_consensus.cc:1261] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240501 14:03:14.211020 15865 pending_rounds.cc:76] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 6
I20240501 14:03:14.211088 15865 pending_rounds.cc:97] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240501 14:03:14.211176 15865 raft_consensus.cc:2866] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.211642 15999 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:14.211853 18400 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.212332 18400 consensus_queue.cc:1035] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.212495 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.212697 15737 raft_consensus.cc:3032] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.212775 15737 raft_consensus.cc:729] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.213047 15737 consensus_queue.cc:260] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.213281 15737 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.214401 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.214982 15999 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.215117 15999 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.215755 15849 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.216614 15737 raft_consensus.cc:2396] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:14.220141 16125 raft_consensus.cc:1261] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 14:03:14.221092 18401 consensus_queue.cc:1035] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.221910 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.222102 16127 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.224110 15997 raft_consensus.cc:1226] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 7. Current term is 8. Ops: []
I20240501 14:03:14.225508 16128 raft_consensus.cc:1226] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 8. Current term is 9. Ops: [2.4-8.6]
I20240501 14:03:14.226110 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.227375 15999 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.227728 16000 raft_consensus.cc:1226] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 8. Current term is 9. Ops: []
I20240501 14:03:14.229195 15998 raft_consensus.cc:1226] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 8. Current term is 9. Ops: []
I20240501 14:03:14.227145 18273 consensus_queue.cc:1046] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.229552 18273 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.229646 18273 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.229913 18273 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.230207 18273 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.228583 18401 consensus_queue.cc:1046] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.234970 18370 raft_consensus.cc:3032] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.235127 18370 raft_consensus.cc:729] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.235492 18370 consensus_queue.cc:260] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.235898 18370 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.237447 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.239250 17643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.240399 16125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.241451 15999 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:14.242486 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.242785 15999 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.242941 15999 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.246017 15999 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.252408 16125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.256279 15999 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.256661 15865 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240501 14:03:14.256745 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.257033 15734 raft_consensus.cc:3032] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.257160 15734 raft_consensus.cc:729] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.257503 15865 pending_rounds.cc:76] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 6
I20240501 14:03:14.257604 15865 pending_rounds.cc:97] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240501 14:03:14.257476 15734 consensus_queue.cc:260] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.257706 15865 raft_consensus.cc:2866] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.257871 15734 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.257843 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.258141 15999 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.258369 15999 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.258586 15998 raft_consensus.cc:1261] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:14.259093 18370 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.259719 18273 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.259876 18273 consensus_queue.cc:1225] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Peer 09ee1930dbed4ac897d146bf885d0e87 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 14:03:14.260764 16000 pending_rounds.cc:76] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Aborting all ops after (but not including) 4
I20240501 14:03:14.260854 16000 pending_rounds.cc:97] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 14:03:14.260952 16000 raft_consensus.cc:2866] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.267175 15998 raft_consensus.cc:1261] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 14:03:14.267652 18370 consensus_queue.cc:1035] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.277038 15737 raft_consensus.cc:2454] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:14.281172 16127 raft_consensus.cc:2396] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:14.281829 16126 raft_consensus.cc:1226] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Rejecting Update request from peer 28461d3f709145e29fc3d3d1c7cce205 for earlier term 7. Current term is 8. Ops: []
I20240501 14:03:14.281674 15934 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.289803 15935 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.291389 18333 raft_consensus.cc:2726] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 14:03:14.291571 18333 raft_consensus.cc:2781] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.291671 18333 raft_consensus.cc:686] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.281201 15999 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.291936 18333 consensus_queue.cc:237] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.293239 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.293460 16126 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.294533 15589 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 6 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.296120 15736 raft_consensus.cc:2454] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:14.296227 16126 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:14.297752 15999 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.299871 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.300185 15999 raft_consensus.cc:482] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.300469 15865 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240501 14:03:14.300599 15865 pending_rounds.cc:76] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 6
I20240501 14:03:14.300671 15865 pending_rounds.cc:97] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240501 14:03:14.300698 15999 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.300774 15865 raft_consensus.cc:2866] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.301676 18401 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.302201 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.302429 15865 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.306777 17910 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.307809 16126 raft_consensus.cc:1261] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:14.308548 15734 raft_consensus.cc:2396] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 14:03:14.308485 18401 consensus_queue.cc:1035] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.312536 15935 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.313488 16000 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 14:03:14.314354 15865 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.314759 15936 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.315165 18337 raft_consensus.cc:2726] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 14:03:14.315335 18337 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.315462 18337 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.315506 15999 raft_consensus.cc:504] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.315765 18337 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.317972 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.318188 15866 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.320807 15999 leader_election.cc:288] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.321926 18401 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.323192 15589 catalog_manager.cc:5484] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 9 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:14.324538 15589 catalog_manager.cc:5484] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 9 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.327226 15869 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240501 14:03:14.327332 15869 pending_rounds.cc:76] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting all ops after (but not including) 6
I20240501 14:03:14.327407 15869 pending_rounds.cc:97] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240501 14:03:14.327463 15869 raft_consensus.cc:2866] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.328661 15866 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.329154 18400 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.329167 15936 leader_election.cc:302] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:14.329586 18333 raft_consensus.cc:2781] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.329766 18333 raft_consensus.cc:686] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.330121 18333 consensus_queue.cc:237] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.330587 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21aad1a0b1740a09ff3774295ad6928" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.330780 15734 raft_consensus.cc:3032] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.330875 15734 raft_consensus.cc:729] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.332942 15734 consensus_queue.cc:260] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.333364 15734 raft_consensus.cc:3037] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.338519 15734 raft_consensus.cc:2454] T f21aad1a0b1740a09ff3774295ad6928 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.342952 16000 raft_consensus.cc:1261] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 14:03:14.344462 18401 consensus_queue.cc:1035] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.346276 16130 raft_consensus.cc:1261] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 14:03:14.346925 18273 consensus_queue.cc:1035] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.351672 15997 raft_consensus.cc:1261] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Refusing update from remote peer 28461d3f709145e29fc3d3d1c7cce205: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 14:03:14.354382 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.354729 15999 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.354964 15999 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.359882 18401 consensus_queue.cc:1035] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.375595 15999 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.375569 15589 catalog_manager.cc:5484] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 6 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.381865 15999 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.382181 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.382450 16127 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.382453 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.382668 15734 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.382761 15734 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.383030 15734 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.383424 15734 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.385393 16127 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:14.385890 15934 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.386170 15734 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 9.
I20240501 14:03:14.386658 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.386886 15999 raft_consensus.cc:482] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.387004 15999 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.387828 15589 catalog_manager.cc:5484] T f21aad1a0b1740a09ff3774295ad6928 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 7 to 8, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:14.389179 15999 raft_consensus.cc:504] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.389883 15999 leader_election.cc:288] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.390149 18337 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.390303 18337 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.390642 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.390847 15865 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.391165 18337 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.393996 15865 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.394438 15936 leader_election.cc:302] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:14.395213 18337 raft_consensus.cc:2781] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.395355 18337 raft_consensus.cc:686] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.395625 18337 consensus_queue.cc:237] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.395985 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b01b2fe30b043fdb721ead3019caeea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.396196 15734 raft_consensus.cc:3032] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.396287 15734 raft_consensus.cc:729] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.396687 15734 consensus_queue.cc:260] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.397186 15734 raft_consensus.cc:3037] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.400043 15734 raft_consensus.cc:2454] T 4b01b2fe30b043fdb721ead3019caeea P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.401149 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.401413 15999 raft_consensus.cc:482] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.401582 15999 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.401343 15589 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 9 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.404500 15999 raft_consensus.cc:504] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.406277 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.406526 15865 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.407575 15592 catalog_manager.cc:5484] T 4b01b2fe30b043fdb721ead3019caeea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 7 to 8, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.409606 15865 raft_consensus.cc:2396] T dc17177ae8b144c0ac3483d8a3540456 P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:14.410393 15999 leader_election.cc:288] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.410053 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17177ae8b144c0ac3483d8a3540456" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.411298 15734 raft_consensus.cc:3032] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.411422 15734 raft_consensus.cc:729] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.412074 15734 consensus_queue.cc:260] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.412401 15734 raft_consensus.cc:3037] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.413149 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.413429 15999 raft_consensus.cc:482] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.413589 15999 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.415277 15734 raft_consensus.cc:2396] T dc17177ae8b144c0ac3483d8a3540456 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:14.415745 15935 leader_election.cc:302] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87; no voters: 28461d3f709145e29fc3d3d1c7cce205, 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:14.416224 18337 raft_consensus.cc:2726] T dc17177ae8b144c0ac3483d8a3540456 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 14:03:14.416543 15999 raft_consensus.cc:504] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.417654 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.417870 16127 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.417990 15999 leader_election.cc:288] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.418977 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.419171 15998 raft_consensus.cc:482] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.419265 15998 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.419992 16127 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.420475 15934 leader_election.cc:302] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.420831 18337 raft_consensus.cc:2781] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.420990 18337 raft_consensus.cc:686] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.421283 18337 consensus_queue.cc:237] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.422184 15998 raft_consensus.cc:504] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.424957 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.425210 15865 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.424855 15589 catalog_manager.cc:5484] T 7042e84f9fdb4a7c81fb314d14a5d29e P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 7 to 8, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:14.428068 15865 raft_consensus.cc:2454] T 4a80377b51164fc5a0b6a6ed342488ea P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 7.
I20240501 14:03:14.428547 15936 leader_election.cc:302] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters:
I20240501 14:03:14.428880 18454 raft_consensus.cc:2781] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:03:14.429090 18454 raft_consensus.cc:686] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.429411 18454 consensus_queue.cc:237] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.430169 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a80377b51164fc5a0b6a6ed342488ea" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.430406 16127 raft_consensus.cc:3037] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:03:14.428866 15998 leader_election.cc:288] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 7 election: Requested vote from peers d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.418521 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7042e84f9fdb4a7c81fb314d14a5d29e" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.431591 15734 raft_consensus.cc:3032] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.431908 15734 raft_consensus.cc:729] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.432345 15999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.432260 15734 consensus_queue.cc:260] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.432616 15999 raft_consensus.cc:482] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.432765 15734 raft_consensus.cc:3037] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.432825 16127 raft_consensus.cc:2396] T 4a80377b51164fc5a0b6a6ed342488ea P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 14:03:14.432802 15999 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.434422 15865 raft_consensus.cc:1261] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:14.435019 18337 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.435796 15734 raft_consensus.cc:2454] T 7042e84f9fdb4a7c81fb314d14a5d29e P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.436751 15999 raft_consensus.cc:504] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.438129 15999 leader_election.cc:288] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.438491 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.439153 15865 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.439141 15998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6"
dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
from {username='slave'} at 127.0.0.1:51662
I20240501 14:03:14.439205 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c18622bc24146d99b2a779cc4e478fe" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.439328 15998 raft_consensus.cc:482] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.439399 15734 raft_consensus.cc:3032] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.439440 15998 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.439491 15734 raft_consensus.cc:729] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.439734 15734 consensus_queue.cc:260] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.440129 15734 raft_consensus.cc:3037] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.442582 15998 raft_consensus.cc:504] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.442971 15734 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.443308 15998 leader_election.cc:288] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68:42895)
I20240501 14:03:14.443459 15935 leader_election.cc:302] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205; no voters:
I20240501 14:03:14.443737 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.443938 15869 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.445817 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d4827abaa478e9c63f779793ed0c6" candidate_uuid: "09ee1930dbed4ac897d146bf885d0e87" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
I20240501 14:03:14.446120 16127 raft_consensus.cc:3037] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.449254 18337 raft_consensus.cc:2781] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.449476 18337 raft_consensus.cc:686] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.449846 18337 consensus_queue.cc:237] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.450879 16130 raft_consensus.cc:1261] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240501 14:03:14.451536 17846 consensus_queue.cc:1035] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.453981 16127 raft_consensus.cc:2396] T da9d4827abaa478e9c63f779793ed0c6 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ee1930dbed4ac897d146bf885d0e87 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:14.454057 15869 raft_consensus.cc:2454] T da9d4827abaa478e9c63f779793ed0c6 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.454579 15865 raft_consensus.cc:2454] T 7c18622bc24146d99b2a779cc4e478fe P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ee1930dbed4ac897d146bf885d0e87 in term 8.
I20240501 14:03:14.456184 15589 catalog_manager.cc:5484] T 4a80377b51164fc5a0b6a6ed342488ea P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 5 to 7, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 7 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.456548 15936 leader_election.cc:302] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, 42389da9ed2d4970ab94df10f7a2cb1b; no voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba
I20240501 14:03:14.456830 17846 raft_consensus.cc:2781] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.457048 17846 raft_consensus.cc:686] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Leader. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.457355 17846 consensus_queue.cc:237] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.459637 16127 raft_consensus.cc:1261] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 14:03:14.460122 18337 consensus_queue.cc:1035] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.460409 18337 consensus_queue.cc:1225] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Peer d5b6839d9f5f4f6ab63bb9aabe75f4ba log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 14:03:14.461299 16127 pending_rounds.cc:76] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 4
I20240501 14:03:14.461520 16127 pending_rounds.cc:97] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 14:03:14.461751 16127 raft_consensus.cc:2866] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.462272 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.465663 15589 catalog_manager.cc:5484] T 7c18622bc24146d99b2a779cc4e478fe P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 7 to 8, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.466346 15589 catalog_manager.cc:5484] T da9d4827abaa478e9c63f779793ed0c6 P 09ee1930dbed4ac897d146bf885d0e87 reported cstate change: term changed from 6 to 8, leader changed from d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68) to 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67). New cstate: current_term: 8 leader_uuid: "09ee1930dbed4ac897d146bf885d0e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:14.467980 15734 raft_consensus.cc:1261] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240501 14:03:14.468041 16129 raft_consensus.cc:1261] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 14:03:14.468103 15734 pending_rounds.cc:76] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 5
I20240501 14:03:14.468175 15734 pending_rounds.cc:97] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240501 14:03:14.468289 15734 raft_consensus.cc:2866] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.469051 17846 consensus_queue.cc:1035] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.469491 17846 consensus_queue.cc:1225] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Peer 28461d3f709145e29fc3d3d1c7cce205 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:14.470141 18337 consensus_queue.cc:1035] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.470325 18337 consensus_queue.cc:1225] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Peer d5b6839d9f5f4f6ab63bb9aabe75f4ba log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 14:03:14.470628 15736 pending_rounds.cc:76] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Aborting all ops after (but not including) 3
I20240501 14:03:14.470718 15736 pending_rounds.cc:97] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:03:14.470818 15736 raft_consensus.cc:2866] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.470911 15736 pending_rounds.cc:97] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 14:03:14.470989 15736 raft_consensus.cc:2866] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.471203 16129 pending_rounds.cc:76] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting all ops after (but not including) 3
I20240501 14:03:14.471283 16129 pending_rounds.cc:97] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 14:03:14.471378 16129 raft_consensus.cc:2866] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.471457 16129 pending_rounds.cc:97] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240501 14:03:14.471527 16129 raft_consensus.cc:2866] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:03:14.477555 15865 raft_consensus.cc:1261] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Refusing update from remote peer 09ee1930dbed4ac897d146bf885d0e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 14:03:14.478085 18337 consensus_queue.cc:1035] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.490641 16126 raft_consensus.cc:482] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.490833 16126 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.492821 16126 raft_consensus.cc:504] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.495929 16126 leader_election.cc:288] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.496009 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.496243 15999 raft_consensus.cc:3032] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.496340 15999 raft_consensus.cc:729] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.496285 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5795defe0a42b68392fc5c6a28098a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.496531 15869 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.496621 15999 consensus_queue.cc:260] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.497587 15999 raft_consensus.cc:3037] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.497787 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.498252 16127 raft_consensus.cc:482] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.498364 16127 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.500303 15869 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.500653 15999 raft_consensus.cc:2454] T ff5795defe0a42b68392fc5c6a28098a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.500923 16066 leader_election.cc:302] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.502028 16127 raft_consensus.cc:504] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.502579 16127 leader_election.cc:288] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.503183 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.503418 15734 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.505697 16126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.505897 16126 raft_consensus.cc:482] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.505996 16126 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.506098 18462 raft_consensus.cc:2781] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.506390 18462 raft_consensus.cc:686] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.506585 15734 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 28461d3f709145e29fc3d3d1c7cce205 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 14:03:14.506893 18462 consensus_queue.cc:237] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.508533 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b6db3827b74bed8016ac9aef243527" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.508652 16126 raft_consensus.cc:504] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.508764 15999 raft_consensus.cc:3032] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 14:03:14.508839 15999 raft_consensus.cc:729] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.509166 15999 consensus_queue.cc:260] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.509442 15999 raft_consensus.cc:3037] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.509832 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.510033 15998 raft_consensus.cc:3032] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.510118 15998 raft_consensus.cc:729] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.510372 15998 consensus_queue.cc:260] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.510708 15998 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.512589 15999 raft_consensus.cc:2396] T b2b6db3827b74bed8016ac9aef243527 P 09ee1930dbed4ac897d146bf885d0e87 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 14:03:14.513319 15998 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.513717 16065 leader_election.cc:302] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:14.513870 16126 leader_election.cc:288] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.514156 16065 leader_election.cc:302] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.514468 18453 raft_consensus.cc:2726] T b2b6db3827b74bed8016ac9aef243527 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240501 14:03:14.514626 18453 raft_consensus.cc:2781] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.514776 18453 raft_consensus.cc:686] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.514839 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433aab52303a4ae5af6a65cc8a97481a" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.515089 15865 raft_consensus.cc:3037] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.515115 18453 consensus_queue.cc:237] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.516600 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.516767 16127 raft_consensus.cc:482] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.516877 16127 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.517263 15865 raft_consensus.cc:2454] T 433aab52303a4ae5af6a65cc8a97481a P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.519395 15592 catalog_manager.cc:5484] T ff5795defe0a42b68392fc5c6a28098a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 7 to 8, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:14.519752 16127 raft_consensus.cc:504] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.520880 16127 leader_election.cc:288] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.520851 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.521114 15734 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.522019 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.522202 16127 raft_consensus.cc:482] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.522286 16127 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.523751 15734 raft_consensus.cc:2454] T e8d8635d7c254bc5ae37c535e66b6dc9 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.524252 16065 leader_election.cc:302] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.524478 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d8635d7c254bc5ae37c535e66b6dc9" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.524727 15865 raft_consensus.cc:3037] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.525768 15592 catalog_manager.cc:5484] T 433aab52303a4ae5af6a65cc8a97481a P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 8 to 9, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 9 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.526050 16127 raft_consensus.cc:504] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.524544 18462 raft_consensus.cc:2781] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.526507 18462 raft_consensus.cc:686] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.527429 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.527385 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d528abe1604862acc3bffe7fbb35a4" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.527627 15869 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.527699 15865 raft_consensus.cc:2396] T e8d8635d7c254bc5ae37c535e66b6dc9 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:14.527674 15998 raft_consensus.cc:3032] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.527772 15998 raft_consensus.cc:729] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.528146 15998 consensus_queue.cc:260] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.528532 15998 raft_consensus.cc:3037] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.529973 15869 raft_consensus.cc:2454] T b7d528abe1604862acc3bffe7fbb35a4 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.530337 16066 leader_election.cc:302] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.530661 17483 raft_consensus.cc:2781] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.530823 17483 raft_consensus.cc:686] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.531080 17483 consensus_queue.cc:237] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.532317 15998 raft_consensus.cc:2396] T b7d528abe1604862acc3bffe7fbb35a4 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:14.533129 16127 leader_election.cc:288] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.531075 18462 consensus_queue.cc:237] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.534682 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.534941 16127 raft_consensus.cc:482] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.535105 16127 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.539135 16127 raft_consensus.cc:504] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.541671 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.541927 15998 raft_consensus.cc:3032] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 14:03:14.542026 15998 raft_consensus.cc:729] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.542222 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160cbc940ba411dbede72e71bc9c851" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.542321 15998 consensus_queue.cc:260] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.542447 15734 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.542726 15998 raft_consensus.cc:3037] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.545497 15998 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P 09ee1930dbed4ac897d146bf885d0e87 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 10.
I20240501 14:03:14.545878 16065 leader_election.cc:302] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ee1930dbed4ac897d146bf885d0e87, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.545740 15734 raft_consensus.cc:2454] T 5160cbc940ba411dbede72e71bc9c851 P 28461d3f709145e29fc3d3d1c7cce205 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 10.
I20240501 14:03:14.546558 16127 leader_election.cc:288] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.547148 15592 catalog_manager.cc:5484] T e8d8635d7c254bc5ae37c535e66b6dc9 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.547544 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.547739 16127 raft_consensus.cc:482] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.547839 16127 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.547722 15592 catalog_manager.cc:5484] T b7d528abe1604862acc3bffe7fbb35a4 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 7 to 8, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.549135 18462 raft_consensus.cc:2781] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Leader election won for term 10
I20240501 14:03:14.549448 18462 raft_consensus.cc:686] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.549769 18462 consensus_queue.cc:237] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.550127 16127 raft_consensus.cc:504] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.551247 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.551506 15734 raft_consensus.cc:3032] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 14:03:14.551582 15734 raft_consensus.cc:729] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 28461d3f709145e29fc3d3d1c7cce205, State: Running, Role: LEADER
I20240501 14:03:14.551785 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb17c062d6546f4ab0f7c79cabcaa91" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.551844 15734 consensus_queue.cc:260] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.552058 15869 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.552131 15734 raft_consensus.cc:3037] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.554368 15734 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.554764 16065 leader_election.cc:302] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.554608 16127 leader_election.cc:288] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.555130 15869 raft_consensus.cc:2454] T 9bb17c062d6546f4ab0f7c79cabcaa91 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.555590 18462 raft_consensus.cc:2781] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.555922 18462 raft_consensus.cc:686] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.556229 18462 consensus_queue.cc:237] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.558028 15592 catalog_manager.cc:5484] T 5160cbc940ba411dbede72e71bc9c851 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 9 to 10, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 10 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:14.561998 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.562252 16127 raft_consensus.cc:482] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.562656 16127 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.563673 15592 catalog_manager.cc:5484] T 9bb17c062d6546f4ab0f7c79cabcaa91 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 7 to 8, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } }
I20240501 14:03:14.565696 16127 raft_consensus.cc:504] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.566653 16127 leader_election.cc:288] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.566879 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.567127 15734 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.567203 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa0651edf94361bcc1efdcd6a2c7bb" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.567415 15869 raft_consensus.cc:3037] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.569624 15734 raft_consensus.cc:2454] T 65fa0651edf94361bcc1efdcd6a2c7bb P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.569876 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.570283 15869 raft_consensus.cc:2396] T 65fa0651edf94361bcc1efdcd6a2c7bb P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 14:03:14.570067 16065 leader_election.cc:302] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.572717 18462 raft_consensus.cc:2781] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.573047 18462 raft_consensus.cc:686] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.573360 18462 consensus_queue.cc:237] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.574267 16127 raft_consensus.cc:482] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.574422 16127 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.576429 16127 raft_consensus.cc:504] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.577534 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.577757 15998 raft_consensus.cc:3032] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 14:03:14.577829 15998 raft_consensus.cc:729] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.578111 15998 consensus_queue.cc:260] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.578364 15998 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.576959 16127 leader_election.cc:288] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.581367 15998 raft_consensus.cc:2396] T e082a4dae96a481289b923efe77fa87b P 09ee1930dbed4ac897d146bf885d0e87 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 14:03:14.582010 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.582193 16127 raft_consensus.cc:482] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.582309 16127 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.582819 15592 catalog_manager.cc:5484] T 65fa0651edf94361bcc1efdcd6a2c7bb P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } }
I20240501 14:03:14.583997 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e082a4dae96a481289b923efe77fa87b" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.584244 15734 raft_consensus.cc:3037] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.585156 16127 raft_consensus.cc:504] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.586254 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.586520 15998 raft_consensus.cc:3032] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.586597 15998 raft_consensus.cc:729] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.586833 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73309c7bc4644aa972bd08799670ef7" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.586881 15998 consensus_queue.cc:260] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.587045 15869 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.587170 15998 raft_consensus.cc:3037] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.590065 15869 raft_consensus.cc:2454] T c73309c7bc4644aa972bd08799670ef7 P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.590167 15998 raft_consensus.cc:2396] T c73309c7bc4644aa972bd08799670ef7 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 14:03:14.590734 16127 leader_election.cc:288] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.591703 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.591912 16127 raft_consensus.cc:482] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.592015 16127 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.592098 16066 leader_election.cc:302] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:14.592440 18462 raft_consensus.cc:2781] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.592604 18462 raft_consensus.cc:686] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.592903 18462 consensus_queue.cc:237] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.595283 15734 raft_consensus.cc:2454] T e082a4dae96a481289b923efe77fa87b P 28461d3f709145e29fc3d3d1c7cce205 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 10.
I20240501 14:03:14.595122 16127 raft_consensus.cc:504] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.595784 16065 leader_election.cc:302] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:14.595983 16127 leader_election.cc:288] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.596727 17483 raft_consensus.cc:2781] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Leader election won for term 10
I20240501 14:03:14.596897 17483 raft_consensus.cc:686] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.596925 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.597163 17483 consensus_queue.cc:237] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.597245 15998 raft_consensus.cc:3032] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 14:03:14.597347 15998 raft_consensus.cc:729] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.597723 15998 consensus_queue.cc:260] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } }
I20240501 14:03:14.598075 15998 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.598757 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef97d453b774b988f34151eb73eee78" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.598976 15736 raft_consensus.cc:3037] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Advancing to term 10
I20240501 14:03:14.601361 15998 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 09ee1930dbed4ac897d146bf885d0e87 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:14.601775 15736 raft_consensus.cc:2396] T 6ef97d453b774b988f34151eb73eee78 P 28461d3f709145e29fc3d3d1c7cce205 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:03:14.602248 16065 leader_election.cc:302] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:14.602435 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.602639 16127 raft_consensus.cc:482] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.602756 16127 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.602859 17483 raft_consensus.cc:2726] T 6ef97d453b774b988f34151eb73eee78 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240501 14:03:14.605592 16127 raft_consensus.cc:504] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.606719 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.606933 15869 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.610337 15869 raft_consensus.cc:2396] T 1a762b8c33a143ac93a2263f92d5b714 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 14:03:14.611078 16127 leader_election.cc:288] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Requested vote from peers 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371), 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261)
I20240501 14:03:14.611800 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a762b8c33a143ac93a2263f92d5b714" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.612020 15734 raft_consensus.cc:3037] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.611061 15592 catalog_manager.cc:5484] T c73309c7bc4644aa972bd08799670ef7 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 8 to 9, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 9 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.613168 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.613421 16127 raft_consensus.cc:482] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.613570 16127 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.616595 16127 raft_consensus.cc:504] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.617372 16127 leader_election.cc:288] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.618198 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.618417 15736 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.618518 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce980b7f336e48acb414191b25f92136"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.618732 16127 raft_consensus.cc:482] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.618881 16127 raft_consensus.cc:3037] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.620901 15736 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.621662 16127 raft_consensus.cc:504] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.622576 16065 leader_election.cc:302] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.622927 18462 raft_consensus.cc:2781] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.622895 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.623042 18462 raft_consensus.cc:686] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.623135 15736 raft_consensus.cc:2379] T ce980b7f336e48acb414191b25f92136 P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in current term 9: Already voted for candidate 09ee1930dbed4ac897d146bf885d0e87 in this term.
I20240501 14:03:14.623332 18462 consensus_queue.cc:237] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.624704 16127 leader_election.cc:288] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.624994 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce980b7f336e48acb414191b25f92136" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.625280 15999 raft_consensus.cc:2379] T ce980b7f336e48acb414191b25f92136 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 LEADER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in current term 9: Already voted for candidate 09ee1930dbed4ac897d146bf885d0e87 in this term.
I20240501 14:03:14.625721 16065 leader_election.cc:302] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87, 28461d3f709145e29fc3d3d1c7cce205
I20240501 14:03:14.625762 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5e643c5ec54233a680726986053d4f"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.625945 16127 raft_consensus.cc:482] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.626034 16127 raft_consensus.cc:3037] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.626129 17483 raft_consensus.cc:2726] T ce980b7f336e48acb414191b25f92136 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 14:03:14.624617 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e094d2d5234896b21d838bb8aa4b20" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.626596 15998 raft_consensus.cc:3037] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.629542 15998 raft_consensus.cc:2454] T f6e094d2d5234896b21d838bb8aa4b20 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.630174 15592 catalog_manager.cc:5484] T e082a4dae96a481289b923efe77fa87b P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 9 to 10, leader changed from 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 10 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.633225 16127 raft_consensus.cc:504] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.634312 15736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"
I20240501 14:03:14.636070 15589 catalog_manager.cc:5484] T f6e094d2d5234896b21d838bb8aa4b20 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 7 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 9 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:14.637656 16127 leader_election.cc:288] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65:43261), 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773)
I20240501 14:03:14.637436 15736 raft_consensus.cc:2454] T db5e643c5ec54233a680726986053d4f P 28461d3f709145e29fc3d3d1c7cce205 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.640769 16065 leader_election.cc:302] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters:
I20240501 14:03:14.641297 17483 raft_consensus.cc:2781] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.641484 17483 raft_consensus.cc:686] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.641639 15734 raft_consensus.cc:2454] T 1a762b8c33a143ac93a2263f92d5b714 P 28461d3f709145e29fc3d3d1c7cce205 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 8.
I20240501 14:03:14.641752 17483 consensus_queue.cc:237] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } }
I20240501 14:03:14.642047 16065 leader_election.cc:302] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28461d3f709145e29fc3d3d1c7cce205, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 42389da9ed2d4970ab94df10f7a2cb1b
I20240501 14:03:14.642315 18462 raft_consensus.cc:2781] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Leader election won for term 8
I20240501 14:03:14.642488 18462 raft_consensus.cc:686] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.642720 18462 consensus_queue.cc:237] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } }
I20240501 14:03:14.643458 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154ac5f942c449686c338594b42ee66"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.643635 16127 raft_consensus.cc:482] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.644054 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5e643c5ec54233a680726986053d4f" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.644388 15999 raft_consensus.cc:2379] T db5e643c5ec54233a680726986053d4f P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in current term 9: Already voted for candidate 09ee1930dbed4ac897d146bf885d0e87 in this term.
I20240501 14:03:14.646165 15592 catalog_manager.cc:5484] T db5e643c5ec54233a680726986053d4f P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 8 to 9, leader changed from 28461d3f709145e29fc3d3d1c7cce205 (127.22.33.65) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 9 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } }
I20240501 14:03:14.647292 16127 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.652055 16127 raft_consensus.cc:504] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.653100 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "09ee1930dbed4ac897d146bf885d0e87"
I20240501 14:03:14.651822 15592 catalog_manager.cc:5484] T 1a762b8c33a143ac93a2263f92d5b714 P d5b6839d9f5f4f6ab63bb9aabe75f4ba reported cstate change: term changed from 6 to 8, leader changed from 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66) to d5b6839d9f5f4f6ab63bb9aabe75f4ba (127.22.33.68). New cstate: current_term: 8 leader_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } } peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } }
I20240501 14:03:14.653334 15999 raft_consensus.cc:3032] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 14:03:14.653419 15999 raft_consensus.cc:729] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 09ee1930dbed4ac897d146bf885d0e87, State: Running, Role: LEADER
I20240501 14:03:14.653693 15999 consensus_queue.cc:260] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.653980 15999 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.657006 15999 raft_consensus.cc:2396] T 9154ac5f942c449686c338594b42ee66 P 09ee1930dbed4ac897d146bf885d0e87 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 14:03:14.658586 16127 leader_election.cc:288] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Requested vote from peers 09ee1930dbed4ac897d146bf885d0e87 (127.22.33.67:44773), 42389da9ed2d4970ab94df10f7a2cb1b (127.22.33.66:40371)
I20240501 14:03:14.659052 15869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154ac5f942c449686c338594b42ee66" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "42389da9ed2d4970ab94df10f7a2cb1b"
I20240501 14:03:14.659281 15869 raft_consensus.cc:3037] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 8 FOLLOWER]: Advancing to term 9
I20240501 14:03:14.659785 16127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c"
dest_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba"
from {username='slave'} at 127.0.0.1:44460
I20240501 14:03:14.660022 16127 raft_consensus.cc:482] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:14.660171 16127 raft_consensus.cc:3037] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 7 FOLLOWER]: Advancing to term 8
I20240501 14:03:14.662228 15869 raft_consensus.cc:2454] T 9154ac5f942c449686c338594b42ee66 P 42389da9ed2d4970ab94df10f7a2cb1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b6839d9f5f4f6ab63bb9aabe75f4ba in term 9.
I20240501 14:03:14.662721 16066 leader_election.cc:302] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42389da9ed2d4970ab94df10f7a2cb1b, d5b6839d9f5f4f6ab63bb9aabe75f4ba; no voters: 09ee1930dbed4ac897d146bf885d0e87
I20240501 14:03:14.663235 18462 raft_consensus.cc:2781] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 FOLLOWER]: Leader election won for term 9
I20240501 14:03:14.663172 16127 raft_consensus.cc:504] T 2b3735e79e334359b05c013508d99f3c P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28461d3f709145e29fc3d3d1c7cce205" member_type: VOTER last_known_addr { host: "127.22.33.65" port: 43261 } } peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.663389 18462 raft_consensus.cc:686] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [term 9 LEADER]: Becoming Leader. State: Replica: d5b6839d9f5f4f6ab63bb9aabe75f4ba, State: Running, Role: LEADER
I20240501 14:03:14.663654 18462 consensus_queue.cc:237] T 9154ac5f942c449686c338594b42ee66 P d5b6839d9f5f4f6ab63bb9aabe75f4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" member_type: VOTER last_known_addr { host: "127.22.33.68" port: 42895 } } peers { permanent_uuid: "09ee1930dbed4ac897d146bf885d0e87" member_type: VOTER last_known_addr { host: "127.22.33.67" port: 44773 } } peers { permanent_uuid: "42389da9ed2d4970ab94df10f7a2cb1b" member_type: VOTER last_known_addr { host: "127.22.33.66" port: 40371 } }
I20240501 14:03:14.664220 15734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3735e79e334359b05c013508d99f3c" candidate_uuid: "d5b6839d9f5f4f6ab63bb9aabe75f4ba" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "28461d3f709145e29fc3d3d1c7cce205"